Autonomic error recovery for a data breakout appliance at the edge of a mobile data network

ABSTRACT

A mechanism provides autonomic recovery for a breakout appliance at the edge of a mobile data network from a variety of errors using a combination of hardware, software and network recovery actions. The recovery actions proceed upon a sliding scale depending on the severity of the problem to achieve the goals of minimizing disruption to traffic flowing through the NodeB while also maintaining an acceptable cost of ownership/maintenance of the system by automatically recovering from as many problems as possible. The error recovery functions within the breakout system hide the error recovery complexities from the management system upstream in the mobile data network. For critical, non-recoverable errors, the autonomic recovery mechanism works in conjunction with a fail-to-wire module to remove the breakout system in the event of a failure in such a way that the mobile data network functions as if the breakout system is no longer present.

BACKGROUND

1. Technical Field

This disclosure generally relates to mobile data networks and morespecifically to autonomic recovery of a data breakout appliance at theedge of a mobile data network.

2. Background Art

Mobile phones have evolved into “smart phones” that allow a user notonly to make a call, but also to access data, such as e-mails, theinternet, etc. Mobile phone networks have evolved as well to provide thedata services that new mobile devices require. For example, 3G networkscover most of the United States, and allow users high-speed wirelessdata access on their mobile devices. In addition, phones are not theonly devices that can access mobile data networks. Many mobile phonecompanies provide equipment and services that allow a subscriber to pluga mobile access card into a Universal Serial Bus (USB) port on a laptopcomputer, and provide wireless internet to the laptop computer throughthe mobile data network. As time marches on, the amount of data servedon mobile data networks will continue to rise exponentially.

Mobile data networks include very expensive hardware and software, soupgrading the capability of existing networks is not an easy thing todo. It is not economically feasible for a mobile network provider tosimply replace all older equipment with new equipment due to the expenseof replacing the equipment. For example, the next generation wirelessnetwork in the United States is the 4G network. Many mobile data networkproviders are still struggling to get their entire system upgraded toprovide 3G data services. Immediately upgrading to 4G equipment is notan economically viable option for most mobile data network providers. Inmany locations, portions of the mobile data network are connectedtogether by point to point microwave links. These microwave links havelimited bandwidth. To significantly boost the throughput of these linksrequires the microwave links to be replaced with fiber optic cable butthis option is very costly.

To facilitate additional capacity on mobile networks, a new “edgeserver” or “breakout system” is being developed by InternationalBusiness Machines Corporation (IBM). The breakout system or edge serveris also referred to as a Mobile Internet Optimization Platform (MIOP).The MIOP component corresponding to each basestation is referred to as aMIOP@NodeB. The MIOP@NodeB offloads (or breaks out) data streams such asinternet data streams for at the edge processing while passing throughthe voice streams to the backend of the network. As used herein, theterm “breakout system” in general means a system that connects betweentwo computer systems on a data network and passes on some of the data onthe data network between the two systems while breaking out for localprocessing other data streams normally flowing between the two computersystems on the data network. A breakout system could broadly beconstrued as a network processing device or mechanism capable of routingall or part of the network traffic on a network data path between twoother nodes through itself.

BRIEF SUMMARY

An autonomic recovery mechanism provides autonomic recovery for abreakout appliance at the edge of a mobile data network from a varietyof errors using a combination of hardware, software and network recoveryactions. The recovery actions proceed upon a sliding scale depending onthe severity of the problem to achieve the dual goals of minimizingdisruption to traffic flowing through the NodeB while also maintainingan acceptable cost of ownership/maintenance of the system byautomatically recovering from as many problems as possible. The errorrecovery functions within the breakout system hide the error recoverycomplexities from the management system upstream in the mobile datanetwork. For critical, non-recoverable errors, the autonomic recoverymechanism works in conjunction with a fail-to-wire (FTW) module toremove the breakout system in the event of a failure in such a way thatthe mobile data network functions as if the breakout system is no longerpresent.

The foregoing and other features and advantages will be apparent fromthe following more particular description, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The disclosure will be described in conjunction with the appendeddrawings, where like designations denote like elements, and:

FIG. 1 is a block diagram of a prior art mobile data network;

FIG. 2 is a block diagram of a mobile data network that includes first,second and third service mechanisms that all communicate via an overlaynetwork;

FIG. 3 is a block diagram of one possible implementation for parts ofthe mobile data network shown in FIG. 2 to illustrate the overlaynetwork;

FIG. 4 is a block diagram of the MIOP@NodeB shown in FIG. 2, whichincludes a first service mechanism;

FIG. 5 is a block diagram of the MIOP@RNC shown in FIG. 2, whichincludes a second service mechanism;

FIG. 6 is a block diagram of the MIOP@Core shown in FIG. 2, whichincludes a third service mechanism;

FIG. 7 is a block diagram of a management mechanism coupled to theoverlay network that manages the functions of MIOP@NodeB, MIOP@RNC, andMIOP@Core;

FIG. 8 is a flow diagram of a method performed by MIOP@NodeB shown inFIGS. 2 and 4;

FIG. 9 is a block diagram showing breakout criteria MIOP@RNC may use inmaking a decision of whether or not to break out data;

FIG. 10 is a flow diagram of a method for the MIOP@NodeB and MIOP@RNC todetermine when to break out data;

FIG. 11 is a flow diagram of a method for the first service mechanism inMIOP@NodeB to selectively break out data when break out for a specifiedsubscriber session has been authorized;

FIG. 12 is a flow diagram of a method for determining when to run MIOPservices for a specified subscriber session;

FIGS. 13-15 are flow diagrams that each show communications between MIOPcomponents when MIOP services are running; and

FIG. 16 is a flow diagram of a method for managing and adjusting theMIOP components;

FIG. 17 is a block diagram of one specific implementation for MIOP@NodeBand MIOP@RNC;

FIGS. 18 and 19 show a flow diagram of a first method for the specificimplementation shown in FIG. 17;

FIG. 20 is a flow diagram of a second method for the specificimplementation shown in FIG. 17;

FIG. 21 is a flow diagram of a third method for the specificimplementation shown in FIG. 17;

FIG. 22 is a flow diagram of a method for the specific implementationshown in FIG. 17 to process a data request that results in a cache missat MIOP@NodeB;

FIG. 23 is a flow diagram of a method for the specific implementationshown in FIG. 17 to process a data request that results in a cache hitat MIOP@NodeB;

FIG. 24 is a block diagram of one specific hardware architecture forMIOP @NodeB;

FIG. 25 is a block diagram of the system controller shown in FIG. 24;

FIG. 26 is a block diagram of the service processor shown in FIG. 24;

FIG. 27 is a block diagram of the security subsystem shown in FIG. 24;

FIG. 28 is a block diagram of the telco breakout system shown in FIG.24;

FIG. 29 is a block diagram of the edge application mechanism 2530 shownin FIG. 25 that performs multiple services at the edge of a mobile datanetwork based on data broken-out at the edge of the mobile data network;

FIG. 30 is a block diagram of the appliance mechanism 2510 shown in FIG.25 that provides interfaces for communicating with MIOP@NodeB;

FIG. 31 is a flow diagram of a method for the appliance mechanism;

FIG. 32 is a block diagram of one specific implementation for theconfiguration management 3022 shown in FIG. 30;

FIG. 33 is a block diagram of one specific implementation for theperformance management 3024 shown in FIG. 30;

FIG. 34 is a block diagram of one specific implementation for thefault/diagnostic management 3026 shown in FIG. 30;

FIG. 35 is a block diagram of one specific implementation for thesecurity management 3028 shown in FIG. 30;

FIG. 36 is a block diagram of one specific implementation for thenetwork management 3030 shown in FIG. 30;

FIG. 37 is a block diagram of one specific implementation for thebreakout management 3032 shown in FIG. 30;

FIG. 38 is a block diagram of one specific implementation for theappliance platform management 3034 shown in FIG. 30;

FIG. 39 is a block diagram of one specific implementation for the edgeapplication management 3036 shown in FIG. 30;

FIG. 40 is a block diagram of one specific implementation for the alarmmanagement 3038 shown in FIG. 30;

FIG. 41 is a block diagram of one specific implementation for the filetransfer management 3040 shown in FIG. 30;

FIG. 42 is a table showing which commands are defined for the applianceinterfaces;

FIG. 43 is a block diagram of the MIOP@NodeB appliance. [0050] FIG. 30is a block diagram that illustrates the data paths of the fail-to-wiremodule when connected to a breakout system between a downstream computerand an upstream computer;

FIG. 44 is a block diagram that illustrates the data paths of thefail-to-wire module when connected to a breakout system between adownstream computer and an upstream computer;

FIG. 45 is a block diagram that illustrates a high level view of thebasic operation of the fail-to-wire module;

FIG. 46 is a block diagram that illustrates a detailed example of thefail-to-wire module in the mobile data network described herein;

FIG. 47 illustrates a block diagram of an exemplary control architecturefor the FTW module in a breakout system with a health monitor having anautonomic recovery mechanism;

FIG. 48 is a block diagram that illustrates a high level view of theMIOP hierarchy of components;

FIG. 49 is a block diagram that illustrates how the autonomic recoverymechanism deals with the different types of errors;

FIG. 50 is a flow diagram of a method for the autonomic recoverymechanism to deal with errors; and

FIG. 51 is a flow diagram of a method for the autonomic recoverymechanism to attempt recovery actions.

DETAILED DESCRIPTION

The basestations of a mobile data network need to be robust to maintainthe constant flow of data to and from user equipment over the network.These basestations are often in remote locations that are not easy toservice. It is important that any failure of a breakout system not takedown the entire basestation. A failure of the breakout system needs tobe managed such that the breakout system failure, which is tasked toenhance the basestation, will not adversely affect the basestation.

The autonomic recovery mechanism described herein provides autonomicrecovery for the breakout system from a variety of failures using acombination of hardware, software and network recovery actions. Therecovery actions proceed upon a sliding scale depending on the severityof the problem to achieve the dual goals of minimizing disruption totraffic flowing through the NodeB while also maintaining an acceptablecost of ownership/maintenance of the system by automatically recoveringfrom as many problems as possible. The error recovery functions withinthe breakout system hide the error recovery complexities from themanagement system upstream in the mobile data network. For critical,non-recoverable errors, the autonomic recovery mechanism works inconjunction with a fail-to-wire (FTW) module to remove the breakoutsystem in the event of a failure in such a way that the mobile datanetwork functions as if the breakout system is no longer present.

Mobile network services are performed in an appliance in a mobile datanetwork in a way that is transparent to most of the existing equipmentin the mobile data network. The mobile data network includes a radioaccess network and a core network. The appliance in the radio accessnetwork breaks out data coming from a basestation, and performs one ormore mobile network services at the edge of the mobile data networkbased on the broken out data. The appliance has defined interfaces anddefined commands on each interface that allow performing all neededfunctions on the appliance without revealing details regarding thehardware and software used to implement the appliance. This appliancearchitecture allows performing new mobile network services at the edgeof a mobile data network within the infrastructure of an existing mobiledata network.

Referring to FIG. 1, a prior art mobile data network 100 is shown.Mobile data network 100 is representative of known 3G networks. Themobile data network 100 preferably includes a radio access network(RAN), a core network, and an external network, as shown in FIG. 1. Theradio access network includes the tower 120, basestation 122 with itscorresponding NodeB 130, and a radio interface on a radio networkcontroller (RNC) 140. The core network includes a network interface onthe radio network controller 140, the serving node 150, gateway node 160and operator service network 170 (as part of the mobile data network).The external network includes any suitable network. One suitable examplefor an external network is the internet 180, as shown in the specificexample in FIG. 1.

In mobile data network 100, user equipment 110 communicates via radiowaves to a tower 120. User equipment 110 may include any device capableof connecting to a mobile data network, including a mobile phone, atablet computer, a mobile access card coupled to a laptop computer, etc.The tower 120 communicates via network connection to a basestation 122.Each basestation 122 includes a NodeB 130, which communicates with thetower 120 and the radio network controller 140. Note there is a fan-outthat is not represented in FIG. 1. Typically there are tens of thousandsof towers 120. Each tower 120 typically has a corresponding base station122 with a NodeB 130 that communicates with the tower. However, networkcommunications with the tens of thousands of base stations 130 areperformed by hundreds of radio network controllers 140. Thus, each radionetwork controller 140 can service many NodeBs 130 in basestations 122.There may also be other items in the network between the basestation 130and the radio network controller 140 that are not shown in FIG. 1, suchas concentrators (points of concentration) or RAN aggregators thatsupport communications with many basestations.

The radio network controller 140 communicates with the serving node 150.In a typical 3G network, the serving node 150 is an SGSN, which is shortfor Service GPRS Support Node, where GPRS stands for general packetradio service. The serving node 150 mediates access to network resourceson behalf of mobile subscribers and implements the packet schedulingpolicy between different classes of quality of service. It is alsoresponsible for establishing the Packet Data Protocol (PDP) context withthe gateway node 160 for a given subscriber session. The serving node150 is responsible for the delivery of data packets from and to thebasestations within its geographical service area. The tasks of theserving node 150 include packet routing and transfer, mobilitymanagement (attach/detach and location management), logical linkmanagement, and authentication and charging functions. The serving node150 stores location information and user profiles of all subscribersregistered with the serving node 150. Functions the serving node 150typically performs include GPRS tunneling protocol (GTP) tunneling ofpackets, performing mobility management as user equipment moves from onebasestation to the next, and billing user data.

In a typical 3G network, the gateway node 160 is a GGSN, which is shortfor gateway GPRS support node. The gateway node 160 is responsible forthe interworking between the core network and external networks. Fromthe viewpoint of the external networks 180, gateway node 160 is a routerto a sub-network, because the gateway node 160 “hides” the core networkinfrastructure from the external network. When the gateway node 160receives data from an external network (such as internet 180) addressedto a specific subscriber, it forwards the data to the serving node 150serving the subscriber. For inactive subscribers paging is initiated.The gateway node 160 also handles routing packets originated from theuser equipment 110 to the appropriate external network. As anchor pointthe gateway node 160 supports the mobility of the user equipment 110. Inessence, the gateway node 160 maintains routing necessary to tunnel thenetwork packets to the serving node 150 that services a particular userequipment 110.

The gateway node 160 converts the packets coming from the serving node150 into the appropriate packet data protocol (PDP) format (e.g., IP orX.25) and sends them out on the corresponding external network. In theother direction, PDP addresses of incoming data packets from theexternal network 180 are converted to the address of the subscriber'suser equipment 110. The readdressed packets are sent to the responsibleserving node 150. For this purpose, the gateway node 160 stores thecurrent serving node address of the subscriber and his or her profile.The gateway node 160 is responsible for IP address assignment and is thedefault router for the subscriber's user equipment 110. The gateway node160 also performs authentication, charging and subscriber policyfunctions. One example of a subscriber policy function is “fair use”bandwidth limiting and blocking of particular traffic types such as peerto peer traffic. Another example of a subscriber policy function isdegradation to a 2G service level for a prepaid subscriber when theprepaid balance is zero.

A next hop router located in the operator service network (OSN) 170receives messages from the gateway node 160, and routes the trafficeither to the operator service network 170 or via an internet serviceprovider (ISP) towards the internet 180. The operator service network170 typically includes business logic that determines how the subscribercan use the mobile data network 100. The business logic that providesservices to subscribers may be referred to as a “walled garden”, whichrefers to a closed or exclusive set of services provided forsubscribers, including a carrier's control over applications, contentand media on user equipment.

Devices using mobile data networks often need to access an externalnetwork, such as the internet 180. As shown in FIG. 1, when a subscriberenters a request for data from the internet, that request is passed fromthe user equipment 110 to tower 120, to NodeB 130 in basestation 122, toradio network controller 140, to serving node 150, to gateway node 160,to operator service network 170, and to internet 180. When the requesteddata is delivered, the data traverses the entire network from theinternet 180 to the user equipment 110. The capabilities of known mobiledata networks 100 are taxed by the ever-increasing volume of data beingexchanged between user equipment 110 and the internet 180 because alldata between the two have to traverse the entire network.

Some efforts have been made to offload internet traffic to reduce thebackhaul on the mobile data network. For example, some mobile datanetworks include a node called a HomeNodeB that is part of the radioaccess network. Many homes have access to high-speed Internet, such asDirect Subscriber Line (DSL), cable television, wireless, etc. Forexample, in a home with a DSL connection, the HomeNodeB takes advantageof the DSL connection by routing Internet traffic to and from the userequipment directly to the DSL connection, instead of routing theInternet traffic through the mobile data network. While this may be aneffective way to offload Internet traffic to reduce backhaul, theHomeNodeB architecture makes it difficult to provide many mobile networkservices such as lawful interception, mobility, and chargingconsistently with the 3G or 4G mobile data network.

Referring to FIG. 2, a mobile data network 200 includes mechanisms thatprovide various services for the mobile data network in a way that istransparent to most of the existing equipment in the mobile datanetwork. FIG. 2 shows user equipment 110, tower 120, NodeB 130, radionetwork controller 140, serving node 150, gateway node 160, operatorservice node 170, and internet 180, the same as shown in FIG. 1. Theadditions to the mobile data network 200 when compared with the priorart mobile data network 100 in FIG. 1 include the addition of threecomponents that may provide mobile network services in the mobile datanetwork, along with a network management mechanism to manage the threecomponents. The mobile network services are performed by what is calledherein a Mobile Internet Optimization Platform (MIOP), and the mobilenetwork services performed by the Mobile Internet Optimization Platformare referred to herein as MIOP services. The three MIOP components thatprovide these mobile network services are shown in FIG. 2 as MIOP@NodeB210, MIOP@RNC 220 and MIOP@Core 230. A network management system shownas MIOP@NMS 240 manages the overall solution by: 1) managing thefunction of the three MIOP components 210, 220 and 230; 2) determiningwhich MIOP@NodeBs in the system aggregate to which MIOP@RNCs via theoverlay network for performance, fault and configuration management; and3) monitoring performance of the MIOP@NodeBs to dynamically change andconfigure the mobile network services. The MIOP@NodeB 210, MIOP@RNC 220,MIOP@Core 230, MIOP@NMS 240, and the overlay network 250, and any subsetof these, and are referred to herein as MIOP components.

The mobile network services provided by MIOP@NodeB 210, MIOP@RNC 220,and MIOP@Core 230 include any suitable services on the mobile datanetwork, such as data optimizations, RAN-aware services,subscriber-aware services, edge-based application serving, edge-basedanalytics, etc. All mobile network services performed by all ofMIOP@NodeB 210, MIOP@RNC 220, and MIOP@Core 230 are included in the termMIOP services as used herein. In addition to the services being offer inthe MIOP components MIOP@NodeB 210, MIOP@RNC 220, and MIOP@Core 230, thevarious MIOP services could also be provided in a cloud based manner.

MIOP@NodeB 210 includes a first service mechanism and is referred to asthe “edge” based portion of the MIOP solution. MIOP@NodeB 210 resides inthe radio access network and has the ability to intercept all traffic toand from the NodeB 130. MIOP@NodeB 210 preferably resides in the basestation 222 shown by the dotted box in FIG. 2. Thus, all data to andfrom the NodeB 130 to and from the radio network controller 140 isrouted through MIOP@NodeB 210. MIOP@NodeB performs what is referred toherein as breakout of data on the intercepted data stream. MIOP@NodeBmonitors the signaling traffic between NodeB and RNC and on connectionsetup intercepts in particular the setup of the transport layer(allocation of the UDP Port, IP address or AAL2 channel). For registeredsessions the breakout mechanism 410 will be configured in a way that alltraffic belonging to this UDP Port, IP address to AAL2 channel will beforwarded to an data offload function. MIOP@NodeB 210 thus performsbreakout of data by defining a previously-existing path in the radioaccess network for non-broken out data, by defining a new second datapath that did not previously exist in the radio access network forbroken out data, identifying data received from a corresponding NodeB asdata to be broken out, sending the data to be broken out on the seconddata path, and forwarding other data that is not broken out on the firstdata path. The signaling received by MIOP@NodeB 210 from NodeB 130 isforwarded to RNC 140 on the existing network connection to RNC 140, eventhough the data traffic is broken out. Thus, RNC 140 sees the signalingtraffic and knows the subscriber session is active, but does not see theuser data that is broken out by MIOP@NodeB 210. MIOP@NodeB thus performstwo distinct functions depending on the monitored data packets: 1)forward the data packets to RNC 140 for signaling traffic and user datathat is not broken out (including voice calls); and 2) re-route the datapackets for user data that is broken out.

Once MIOP@NodeB 210 breaks out user data it can perform any suitableservice based on the traffic type of the broken out data. Because theservices performed by MIOP@NodeB 210 are performed in the radio accessnetwork (e.g., at the basestation 222), the MIOP@NodeB 210 can servicethe user equipment 110 much more quickly than can the radio networkcontroller 140. In addition, by having a MIOP@NodeB 210 that isdedicated to a particular NodeB 130, one MIOP@NodeB only needs toservice those subscribers that are currently connected via a singleNodeB. The radio network controller, in contrast, which typicallyservices dozens or even hundreds of basestations, must service all thesubscribers accessing all basestations it controls from a remotelocation. As a result, MIOP@NodeB is in a much better position toprovide services that will improve the quality of service and experiencefor subscribers than is the radio network controller.

Breaking out data in the radio access network by MIOP@NodeB 210 allowsfor many different types of services to be performed in the radio accessnetwork. These services may include optimizations that are similar tooptimizations provided by known industry solutions between radio networkcontrollers and the serving node. However, moving these optimizations tothe edge of the mobile data network will not only greatly improve thequality of service for subscribers, but will also provide a foundationfor applying new types of services at the edge of the mobile datanetwork, such as terminating machine-to-machine (MTM) traffic at theedge (e.g., in the basestation), hosting applications at the edge, andperforming analytics at the edge.

MIOP@RNC 220 includes a second service mechanism in mobile data network200. MIOP@RNC 220 monitors all communication between the radio networkcontroller 140 and serving node 150. The monitored communications areall communications to and from the radio network controller and the restof the core network. MIOP@RNC 220 may provide one or more services forthe mobile data network. MIOP@RNC 220 preferably makes the decision ofwhether or not to allow breakout of data. If MIOP@RNC 220 decides tobreakout data for a given subscriber session, it may send a message toMIOP@NodeB 210 authorizing breakout by MIOP@NodeB 210, or may decide tobreakout the data at MIOP@RNC 220, depending on the configured breakoutdecision criteria and selected radio channel. Because messages to andfrom the core network establishing the PDP context for a givensubscriber session are monitored by MIOP@RNC 220, the decision ofwhether or not to breakout data resides in the MIOP@RNC 220.

MIOP@Core 230 includes a third service mechanism in the mobile datanetwork 200. MIOP@Core 230 may include all the same services as MIOP@RNC220, or any suitable subset of those services. If the decision is madenot to provide services at MIOP@NodeB 210 or MIOP@RNC 220, these sameservices plus more sophisticated services can be performed at MIOP@Core230. Thus, mobile data network 200 provides flexibility by allowing adecision to be made of where to perform which services. BecauseMIOP@NodeB 210, MIOP@RNC 220 and MIOP@Core 230 preferably include someof the same services, the services between components may interact(e.g., MIOP@NodeB and MIOP@Core may interact to optimize TCP trafficbetween them), or the services may be distributed across the mobile datanetwork (e.g., MIOP@NodeB performs breakout and provides services forhigh-speed traffic, MIOP@RNC performs breakout and provides services forlow-speed traffic, and MIOP@Core provides services for non-broken outtraffic). The MIOP system architecture thus provides a very powerful andflexible solution, allowing dynamic configuring and reconfiguring on thefly of which services are performed by the MIOP components and where. Inaddition, these services may be implemented taking advantage of existinginfrastructure in a mobile data network.

MIOP@NMS 240 is a network management system that monitors and controlsthe functions of MIOP@NodeB 210, MIOP@RNC 220, and MIOP@Core 230.MIOP@NMS 240 preferably includes MIOP internal real-time or nearreal-time performance data monitoring to determine if historical oradditional regional dynamic changes are needed to improve services onthe mobile data network 200. MIOP@NMS 240 provides a user interface thatallows a system administrator to operate and to configure how the MIOPcomponents 210, 220 and 230 function.

The overlay network 250 allows MIOP@NodeB 210, MIOP@RNC 220, MIOP@Core230, and MIOP@NMS 240 to communicate with each other. The overlaynetwork 250 is preferably a virtual private network primarily on anexisting physical network in the mobile data network. Thus, whileoverlay network 250 is shown in FIG. 2 separate from other physicalnetwork connections, this representation in FIG. 2 is a logicalrepresentation.

FIG. 3 shows one suitable implementation of a physical network and theoverlay network in a sample mobile data system. The existing physicalnetwork in the mobile data network before the addition of the MIOP@NodeB210, MIOP@RNC 220, and MIOP@Core 230 is shown by the solid lines witharrows. This specific example in FIG. 3 includes many NodeBs, shown inFIG. 1 as 130A, 130B, 130C, . . . , 130N. Some of the NodeBs have acorresponding MIOP@NodeB. FIG. 3 illustrates that MIOP@NodeBs (such as210A and 210N) can be placed in a basestation with its correspondingNodeB, or can be placed upstream in the network after a point ofconcentration (such as 210A after POC3 310). FIG. 3 also illustratesthat a single MIOP@NodeB such as MIOP@NodeB1 210A can service twodifferent NodeBs, such as NodeB1 130A and NodeB2 130B. Part of theoverlay network is shown by the dotted lines between MIOP@NodeB1 210Aand second point of concentration POC2 320, between MIOP@NodeB3 210C andPOC3 315, between MIOP@NodeBN 210N and POC3 315, and between POC3 315and POC2 320. Note the overlay network in the radio access networkportion is a virtual private network that is implemented on the existingphysical network connections. The overlay network allows the MIOP@NodeBs210A, 210C and 210N to communicate with each other directly, which makessome services possible in the mobile data network 200 that werepreviously impossible. FIG. 3 shows MIOP@NodeB1 210A connected to asecond point of concentration POC2 320. The broken arrows coming in fromabove at POC2 320 represent connections to other NodeBs, and could alsoinclude connections to other MIOP@NodeBs. Similarly, POC2 320 isconnected to a third point of concentration POC1 330, with possiblyother NodeBs or MIOP@NodeBs connected to POC1. The RNC 140 is shownconnected to POC1 330, and to a first router RT1 340 in the corenetwork. The router RT1 340 is also connected to the SGSN 150. While notshown in FIG. 3 for the sake of simplicity, it is understood that SGSNin FIG. 3 is also connected to the upstream core components shown inFIG. 2, including GGSN 160, OSN 170 and internet 180.

As shown in FIG. 3, the overlay network from the NodeBs to POC1 330 is avirtual private network implemented on existing physical networkconnections. However, the overlay network requires a second router RT2350, which is connected via a physical network connection 360 to POC1330, and is connected via physical network connection 370 to MIOP@RNC220. This second router RT2 350 may be a separate router, or may be arouter implemented within MIOP@RNC 220. MIOP@RNC 220 is also connectedto router RT1 340 via a physical network connection 380, and is alsoconnected to MIOP@Core 230. Physical connection 380 in FIG. 3 is shownin a line with short dots because it is not part of the pre-existingphysical network before adding the MIOP components (arrows with solidlines) and is not part of the overlay network (arrows with long dots).Note the connection from MIOP@RNC 220 to MIOP@Core 230 is via existingphysical networks in the core network.

We can see from the configuration of the physical network and overlaynetwork in FIG. 3 that minimal changes are needed to the existing mobiledata network to install the MIOP components. The most that must be addedis one new router 350 and three new physical network connections 360,370 and 380. Once the new router 350 and new physical networkconnections 360, 370 and 380 are installed, the router 350 and MIOPcomponents are appropriately configured, and the existing equipment inthe mobile data network is configured to support the overlay network,the operation of the MIOP components is completely transparent toexisting network equipment.

As can be seen in FIG. 3, data on the overlay network is defined onexisting physical networks from the NodeBs to POC1. From POC1 theoverlay network is on connection 360 to RT2 350, and on connection 370to MIOP@RNC 220. Thus, when MIOP@NodeB 210 in FIG. 2 needs to send amessage to MIOP@RNC 220, the message is sent by sending packets via avirtual private network on the physical network connections to POC1,then to RT2 350, then to MIOP@RNC 220. Virtual private networks arewell-known in the art, so they are not discussed in more detail here.

Referring to FIG. 4, MIOP@NodeB 210 preferably includes a breakoutmechanism 410, an edge service mechanism 430, and an overlay networkmechanism 440. The breakout mechanism 410 determines breakoutpreconditions 420 that, when satisfied, allow breakout to occur at thisedge location. Breakout mechanism 410 in MIOP@NodeB 210 communicateswith the breakout mechanism 510 in MIOP@RNC 220 shown in FIG. 5 to reacha breakout decision. The breakout mechanism 410, after receiving amessage from MIOP@RNC 220 authorizing breakout on connection setupintercepts in particular the setup of the transport layer (allocation ofthe UDP Port, IP address or AAL2 channel). For authorized sessions thebreakout mechanism 410 will be configured in a way that all trafficbelonging to this UDP Port, IP address to AAL2 channel will be forwardedto a data offload function. For traffic that should not be broken out,the breakout mechanism 410 sends the data on the original data path inthe radio access network. In essence, MIOP@NodeB 210 intercepts allcommunications to and from the basestation 130, and can perform services“at the edge”, meaning at the edge of the radio access network that isclose to the user equipment 110. By performing services at the edge, theservices to subscribers may be increased or optimizes without requiringhardware changes to existing equipment in the mobile data network.

The breakout mechanism 410 preferably includes breakout preconditions420 that specify one or more criterion that must be satisfied beforebreakout of data is allowed. One suitable example of breakoutpreconditions is the speed of the channel. In one possibleimplementation, only high-speed channels will be broken out atMIOP@NodeB 210. Thus, breakout preconditions 420 could specify thatsubscribers on high-speed channels may be broken out, while subscriberson low-speed channels are not broken out at MIOP@NodeB 210. When thebreakout preconditions 420 are satisfied, the MIOP@NodeB 210 registersthe subscriber session with MIOP@RNC 220. This is shown in method 800 inFIG. 8. MIOP@NodeB 210 intercepts and monitors network traffic to andfrom NodeB (basestation) (step 810). When the traffic does not satisfythe breakout preconditions (step 820=NO), method 800 returns to step810. When the traffic satisfies the breakout conditions (step 820=YES),MIOP@NodeB 210 sends a message to MIOP@RNC 220 on the overlay network250 to register the subscriber session for breakout (step 830). With thesubscriber session registered with MIOP@RNC 220, the MIOP@RNC 220 willdetermine whether or not to breakout data for the subscriber session,and where the breakout is done, as explained in more detail below.

Referring back to FIG. 4, MIOP@NodeB 210 also includes an edge servicemechanism 430. The edge service mechanism 430 provides one or moreservices for the mobile data network 200. The edge service mechanism 430may include any suitable service for the mobile data network includingwithout limitation caching of data, data or video compressiontechniques, push-based services, charging, application serving,analytics, security, data filtering, new revenue-producing services,etc. The edge service mechanism is the first of three service mechanismsin the MIOP components. While the breakout mechanism 410 and edgeservice mechanism 430 are shown as separate entities in FIG. 4, thefirst service mechanism could include both breakout mechanism 410 andedge service mechanism 430.

MIOP@NodeB 210 also includes an overlay network mechanism 440. Theoverlay network mechanism 440 provides a connection to the overlaynetwork 250 in FIG. 2, thereby allowing MIOP@NodeB 210 to communicatewith MIOP@RNC 220, MIOP@Core 230, and MIOP@NMS 240. As stated above, theoverlay network 250 is preferably a virtual private network primarily onan existing physical network in the mobile data network 200.

Referring to FIG. 5, MIOP@RNC 220 preferably includes a breakoutmechanism 510, an RNC service mechanism 540, an overlay networkmechanism 550, and business intelligence 560. Breakout mechanism 510includes breakout criteria 520 that specifies one or more criterionthat, when satisfied, allows breakout of data. Subscriber registrationmechanism 530 receives messages from MIOP@NodeB 210, and registerssubscriber sessions for which the breakout preconditions 420 inMIOP@NodeB 210 are satisfied. When the breakout mechanism 510 determinesthe breakout criteria 520 is satisfied, the breakout mechanism 510 willthen determine where the breakout should occur. When the breakout canoccur at MIOP@NodeB 210, the MIOP@RNC 220 sends a message to MIOP@NodeB210 on the overlay network 250 authorizing breakout at MIOP@NodeB 210.When the breakout should occur at MIOP@RNC 220, the breakout mechanism510 in MIOP@RNC 220 performs the breakout as well for the trafficremaining then). This is shown in more detail in method 1000 in FIG. 10.MIOP@RNC monitors network traffic between the radio network controller140 and the serving node 150 (step 1010). When the traffic does notsatisfy the breakout criteria (step 1020=NO), method 1000 loops back tostep 1010. When the network traffic satisfies the breakout criteria(step 1020=YES), the breakout mechanism 510 determines whether thesubscriber session is registered for breakout (step 1030). A subscribersession is registered for breakout when the MIOP@NodeB 210 determinedthe traffic satisfied the breakout preconditions and registered thesubscriber session for breakout, as shown in FIG. 8. Returning to FIG.10, when the subscriber is registered for breakout (step 1030=YES),MIOP@RNC 220 sends a message via the overlay network 250 to MIOP@NodeB210 authorizing breakout of traffic for the subscriber session (step1040). MIOP@NodeB 210 may then breakout traffic for the subscribersession (step 1050). When the subscriber is not registered for breakout(step 1030=NO), method 1000 checks to see if MIOP@RNC is going to dobreakout (step 1060). If not (step 1060=N0), method 1000 is done. WhenMIOP@RNC is going to do breakout (step 1060=YES), the traffic is thenbroken out at MIOP@RNC (step 1070).

In one specific example, the breakout preconditions specify onlyhigh-speed channels are broken out at MIOP@NodeB 210, and when thebreakout preconditions are satisfied, the subscriber session isregistered for breakout, as shown in FIG. 8. FIG. 10 illustrates thateven when the breakout preconditions are not satisfied, breakout canstill be performed at MIOP@RNC 220. Thus, even if the subscriber sessionis on a low-speed channel, if all the other breakout criteria aresatisfied, breakout of the low-speed channel may be performed atMIOP@RNC 220. The mobile data network 200 thus provides greatflexibility in determining when to do breakout and where.

Referring back to FIG. 5, the RNC service mechanism 540 provides one ormore services for the mobile data network. RNC service mechanism 540 isthe second of three service mechanisms in the MIOP components. The RNCservice mechanism 540 may include any suitable service for the mobiledata network, including without limitation caching of data, data orvideo compression techniques, push-based services, charging, applicationserving, analytics, security, data filtering, new revenue-producingservices, etc.

While the breakout mechanism 510 and RNC service mechanism 540 are shownas separate entities in FIG. 5, the second service mechanism couldinclude both breakout mechanism 510 and RNC service mechanism 540. Theoverlay network mechanism 550 is similar to the overlay networkmechanism 440 in FIG. 4, providing a logical network connection to theother MIOP components on the overlay network 250 in FIG. 2. MIOP@RNC 220also includes business intelligence 560, which includes:

-   -   1) historical subscriber information received from the mobile        data network over time, such as mobility and location, volumes,        traffic types, equipment used, etc.    -   2) network awareness, including NodeB load states, service area        code, channel type, number of times channel type switching        occurred for a PDP session, serving cell ID, how many cells and        their IDs are in the active set, PDP context type, PDP sessions        per subscriber, session duration, data consumption, list of        Uniform Resource Locators (URLs) browsed for user        classification, top URL browsed, first time or repeat user,        entry point/referral URLs for a given site, session tracking,        etc.    -   3) association of flow control procedures between NodeB and RNC        to subscribers.

The business intelligence 560 may be instrumented by the RNC servicemechanism 540 to determine when and what types of MIOP services toperform for a given subscriber. For example, services for a subscriberon a mobile phone may differ when compared to services for a subscriberusing a laptop computer to access the mobile data network. In anotherexample, voice over internet protocol (VoIP) session could have the databroken out.

Referring to FIG. 6, the MIOP@Core 230 includes a core service mechanism610 and an overlay network mechanism 620. Core service mechanism 610provides one or more services for the mobile data network. Core servicemechanism 610 is the third of three service mechanisms in the MIOPcomponents. The core service mechanism 610 may include any suitableservice for the mobile data network, including without limitationcaching of data, data or video compression techniques, push-basedservices, charging, application serving, analytics, security, datafiltering, new revenue-producing services, etc. In one specificimplementation, the MIOP@Core 230 is an optional component, because allneeded services could be performed at MIOP@NodeB 210 and MIOP@RNC 220.In an alternative implementation, MIOP@Core 230 performs some services,while MIOP@RNC performs others or none. The overlay network mechanism620 is similar to the overlay network mechanisms 440 in FIGS. 4 and 550in FIG. 5, providing a logical network connection to the other MIOPcomponents on the overlay network 250 in FIG. 2.

Referring to FIG. 7, the MIOP@NMS 240 is a network management systemthat monitors and manages performance of the mobile data network 200,and controls the function of MIOP@NodeB 210, MIOP@RNC 220, and MIOP@Core230. MIOP@NMS 240 preferably includes a network monitoring mechanism710, a performance management mechanism 720, a security managementmechanism 730, and a configuration management mechanism 740. The networkmonitoring mechanism 710 monitors network conditions, such as alarms, inthe mobile data network 200. The performance management mechanism 720can enable, disable or refine certain services by supporting theexecution of services in real-time or near real-time, such as servicesthat gather information to assess customer satisfaction. The securitymanagement mechanism 730 manages security issues in the mobile datanetwork, such as intrusion detection or additional data privacy. Theconfiguration management mechanism 740 controls and manages theconfiguration of MIOP@NodeB 210, MIOP@RNC 220, and MIOP@Core 230 in away that allows them to dynamically adapt to any suitable criteria,including data received from the network monitoring mechanism, time ofday, information received from business intelligence 560, etc.

FIG. 9 shows sample breakout criteria 520 shown in FIG. 5 and used instep 1020 in FIG. 10. Suitable breakout criteria 520 includes accesspoint name, user equipment identifier, user equipment type, quality ofservice, subscriber ID, mobile country code, and mobile network code.For example, breakout criteria 520 could specify to perform MIOPservices for the operator's subscribers, and not to perform MIOPservices for roamers. In another example, the breakout criteria 520could specify to break out only video requests. A static breakoutdecision will be performed during PDP Context Activation. Based on IPflows (e.g. shallow packet inspection of the IP 5 tuple) only specificIP flows maybe identified and broken out dynamically within that PDPsubscriber session (e.g., VoIP traffic), as discussed in more detailbelow with respect to FIG. 11. Breakout criteria 520 expressly extendsto any suitable criteria for making the breakout decision.

Referring again to FIG. 10, when the traffic satisfies the breakoutcriteria (step 1020=YES), and the subscriber session is registered forbreakout (step 1030=YES), MIOP@RNC sends a message to MIOP@NodeBauthorizing breakout of traffic for this subscriber session (step 1040).In response, MIOP@NodeB begins decrypting the bearer, examining thesignaling and user IP traffic tunneled through it and may breakout thetraffic for this subscriber session (step 1050). Note, however,MIOP@NodeB may still decide not to breakout all traffic based on othercriteria, such as type of IP request the destination of the traffic orthe ISO Layer 7 Application of the decrypted user traffic. Determinationof the Application may be performed simply by inspection of the IP5-tuple or optionally via inspection at layer 7 using Deep PacketInspection (DPI) techniques. This is shown in the specific example inFIG. 11. Method 1050 in FIG. 10 is one suitable implementation of step1050 in FIG. 10. MIOP@NodeB monitors IP requests from the subscriber(step 1110). When the user traffic IP request matches a specified typecriteria (step 1120=YES), the IP session is broken out for thesubscriber (step 1130). When the IP request does not match a specifiedcriteria type (step 1120=NO), no breakout is performed. For example,let's assume that IP requests to access video over the RTP layer 7Application Protocol are broken out so the video data may be cached inMIOP@NodeB 210, but other requests, such as Google searches, are not.The MIOP@NodeB monitors the IP requests from the subscriber (step 1110),and when the subscriber session IP request carries RTP traffic is for avideo file (step 1120=YES), the IP session is broken out (step 1130).Otherwise, the IP session is not broken out at MIOP@NodeB. This is onesimple example to illustrate additional flexibility and intelligencewithin MIOP@NodeB that may determine whether or not to perform breakoutfor a given subscriber session at the MIOP@NodeB after being authorizedby MIOP@RNC to perform breakout for that subscriber session. Anysuitable criteria could be used to determine what to breakout and whenat MIOP@NodeB once MIOP@NodeB has been authorized for breakout in step1040 in FIG. 10.

Referring to FIG. 12, method 1200 shows a method for determining when torun MIOP services. The Packet Data Protocol (PDP) activation context fora subscriber is monitored (step 1210). A PDP activation context isestablished when user equipment 110 connects to tower 120 and thesubscriber runs an application that triggers the PDP activationprocedure. The core network will determine the subscriber, and perhapscorresponding user equipment. When MIOP services are allowed (step1220=YES), services for this subscriber session are run (step 1230) uponthe arrival of data from the subscriber. When MIOP services are notallowed (step 1220=NO), no MIOP services are run. In one simple example,MIOP services in the mobile data network are allowed for authorizedsubscribers, but are not allowed for subscribers from a differentwireless company that are roaming.

MIOP services may require communicating between MIOP components on theoverlay network. Referring to FIG. 13, a method 1300 showscommunications by MIOP@NodeB when MIOP services are running (step 1310).When the edge service mechanism requires communication with MIOP@RNC(step 1320=YES), MIOP@NodeB exchanges messages with MIOP@RNC over theoverlay network (step 1330). When the edge service mechanism requirescommunication with MIOP@Core (step 1340=YES), MIOP@NodeB exchangesmessages with MIOP@Core over the overlay network (step 1350). Theoverlay network thus allows the various MIOP components to communicatewith each other when MIOP services are running.

FIG. 14 shows a method 1400 that shows communications by MIOP@RNC whenMIOP services are running (step 1410). When the RNC service mechanismrequires communication with MIOP@NodeB (step 1420=YES), MIOP@RNCexchanges messages with MIOP@NodeB over the overlay network (step 1430).When the RNC service mechanism requires communication with MIOP@Core(step 1440=YES), MIOP@RNC exchanges messages with MIOP@Core over theoverlay network (step 1450).

FIG. 15 shows a method 1500 that shows communications by MIOP@Core whenMIOP services are running (step 1510). When the core service mechanismrequires communication with MIOP@NodeB (step 1520=YES), MIOP@Coreexchanges messages with MIOP@NodeB over the overlay network (step 1530)relayed via MIOP@RNC. When the core service mechanism requirescommunication with MIOP@RNC (step 1540=YES), MIOP@Core exchangesmessages with MIOP@RNC over the overlay network (step 1550).

FIG. 16 shows a method 1600 that is preferably performed by MIOP@NMS 240in FIGS. 2 and 7. The performance and efficiency of the MIOP componentsthat perform MIOP services are monitored (step 1610). The MIOPcomponents that perform MIOP services may include MIOP@NodeB 210,MIOP@RNC 220, and MIOP@Core 230, assuming all of these components arepresent in the mobile data network 200. When performance may be improved(step 1620=YES), the performance of the MIOP components is adjusted (ifimplemented and applicable) by sending one or more network messages viathe overlay network (step 1630). Note also a human operator could alsomanually reconfigure the MIOP components to be more efficient.

Referring to FIG. 17, implementations for MIOP@NodeB 210 and MIOP@RNC220 are shown by way of example. Other implementations are possiblewithin the scope of the disclosure and claims herein. User equipment 110is connected to NodeB 130. Note the antenna 120 shown in FIG. 2 is notshown in FIG. 17, but is understood to be present to enable thecommunication between user equipment 110 and NodeB 130. MIOP@NodeB 210includes an edge cache mechanism 1730, which is one suitable example ofedge service mechanism 430 in FIG. 4. MIOP@NodeB 210 includes aninterface referred to herein as IuB Data Offload Gateway (IuB DOGW)1710. This gateway 1710 implements the breakout mechanism 410 accordingto one or more specified breakout preconditions 420 shown in FIG. 4. IuBDOGW 1710 includes a switching application 1740, an offload data handler1750, and an RNC channel handler 1760. The switching application 1740 isresponsible for monitoring data packets received from NodeB 130, thebroken out data packets to the offload data handler forwards accordingto it configuration, relays the non-broken out data packets and controlsystem flows to the RNC 140 via the original connections in the RAN.While switching application 1740 is shown as two separate boxes in FIG.17, this is done to visually indicate the switching application 1740performs switching on two different interfaces, the network interfaceand overlay network interface, but the switching application 1740 ispreferably a single entity.

When a breakout decision is made and MIOP@RNC 220 sends a message toMIOP@NodeB 210 authorizing breakout (see step 1040 in FIG. 10), whenMIOP@NodeB decides to breakout specified user data, the specified userdata received by the switching application 1740 from NodeB 130 is brokenout, which means the switching application 1740 routes the specifieduser data to the offload data handler 1750 so the broken out data isrouted to the data path defined for breakout data. The offload datahandler 1750 may send the data to the edge cache mechanism 1730 forprocessing, which can route the data directly to MIOP@RNC 220 via theoverlay network, as shown by the path with arrows going from NodeB 130to MIOP@RNC 220.

User data that is not broken out and signaling traffic is routeddirectly back by the switching application 1740 to the RNC. In thismanner, non-broken out data and signaling traffic passes through the IuBDOGW 1710 to RNC 140, while broken out data is routed by the IuB DOGW1710 to a different destination. Note that edge cache mechanism 1730 maysend messages to MIOP@RNC 220 as shown in FIG. 17, but the broken outmessages themselves are not sent to MIOP@RNC 220.

MIOP@RNC 220 includes an interface referred to herein as IuPS dataoffload gateway (IuPS DOGW) 1770. IuPS DO GW 1770 forwards all signalingand non-broken out data traffic from RNC 140 to SGSN 150 via the GTPtunnel. IuPS DOGW 1770 includes the breakout mechanism 510, breakoutcriteria 520 and subscriber registration mechanism 530 shown in FIG. 5and discussed above with reference to FIG. 5. IuPS DOGW 1770 mayexchange messages with IuB DOGW 1710 via the overlay network to performany needed service in MIOP@NodeB 210 or MIOP@RNC 220. For the specificimplementation shown in FIG. 17, while the IuPS DOGW 1770 in MIOP@RNC220 does not include an offload data handler, the IuPS DOGW 1770 couldinclude an offload data handler and switching application similar tothose shown in MIOP@NodeB 210 when MIOP@RNC 220 also needs to performbreakout of data.

The IuPS DOGW 1770 includes an RNC channel handler 1780. The RNC channelhandlers 1760 in MIOP@NodeB 210 and 1780 in MIOP@RNC 220 monitor datatraffic to and from RNC 140 related to a broken out subscriber sessionand provide a keep-alive channel maintenance mechanism.

Specific methods are shown in FIGS. 18-21 that illustrate how thespecific implementation in FIG. 17 could be used. FIGS. 18 and 19 show amethod 1800 for setting up breakout of data. The UE sends a connectionrequest to the RNC (step 1810). The RNC sets up a radio link via NodeB(step 1815). The RNC then sets up a network connection with NodeB (step1820). The UE and SGSN then communicate for the attach andauthentication procedure (step 1825). IuB DOGW detects the leadingmessage in the attach and authentication procedure, and registers thesubscriber session with IuPS DOGW when preconditions are fulfilled (e.g.UE is capable to carry high speed traffic) (step 1830). During theattach and authentication procedure, IuPS DOGW monitors the securitycontext sent from SGSN to RNC (step 1835). IuPS DOGW then sends keys toIuB DOGW (step 1840). These keys are needed to decipher (decrypt) theupcoming signaling and uplink user data and to cipher (encrypt) thedownlink user data. UE then requests PDP context activation to SGSN(step 1845). In response, SGSN sets up a network tunnel to RNC (step1850). IuPS DOGW monitors network tunnel setup from SGSN to RNC andmakes a decision breakout=YES (step 1855). IuPS DOGW sends a message toIuB DOGW indicating breakout=YES (step 1860). Continuing on FIG. 19,SGSN sends an RAB assignment request to UE (step 1865). IuPS DOGWdetects the RAB assignment request from SGSN to UE and replaces the SGSNtransport address with IuPS DOGW transport address (step 1870). IuPSDOGW sends a message to MIOP@Core indicating breakout=YES (step 1875).RNC communicates with NodeB and UE to (re) configure signaling and dataradio bearer (step 1880). RNC acknowledges to SGSN when RAB assignmentis complete (step 1885). SGSN accepts PDP context activation by sendinga message to UE (step 1890). UE and SGSN may then exchange data for thePDP context (step 1895).

Referring to FIG. 20, a method 2000 begins by establishing a PDP context(step 2010). Method 1800 in FIGS. 18 and 19 include the detailed stepsfor establishing a PDP context. When breakout=YES, RAB assignmentrequests from SGSN to RNC are monitored by IuPS DOGW (step 2020). IuPSDOGW modifies any RAB assignment requests from SGSN to RNC to replacethe SGSN transport address in the RAB assignment request with the IuPSDOGW transport address (step 2030) in case of matching breakout criteriaduring PDP context activation procedure. The switching application onIuB DOGW is configured upon the RAN transport layer setup to identifybased on IP addresses and ports the broken out traffic and forwards thistraffic to the Offload data handler 1765, and forwards non-broken outtraffic and control system data flows to the RNC (step 2040).

Referring to FIG. 21, a method 2100 begins when NodeB sends data towardsRNC (step 2110). The switching application in IuB DOGW redirects thebroken out traffic to the edge service mechanism (step 2120), such asedge cache mechanism 1730 in FIG. 17. The switching application alsoforwards non-broken out data and signaling data to the RNC (step 2130)via the original RAN connections. The RNC can still receive data fornon-broken out traffic from MIOP@NodeB when breakout=YES (step 2140).The RNC then sends non-broken out traffic from MIOP@NodeB from UE whenbreakout=YES to IuPS DOGW transport address specified in RAB assignmentrequest (step 2150).

A simple example is now provided for the specific implementation in FIG.17 to show how data can be cached and delivered by MIOP@NodeB 210.Referring to FIG. 22, method 2200 represents steps performed in theimplementation in FIG. 17 for a cache miss. UE sends a data request toNodeB (step 2210). NodeB sends the data request to IuB DOGW (step 2215).We assume the requested data meets the offload criteria at MIOP@NodeB(step 2220), which means MIOP@NodeB has been authorized to performbreakout and has determined this requested data should be broken out.IuB DOGW sends the data request to the edge cache mechanism (step 2225).We assume the data is not present in the edge cache mechanism, so due tothe cache miss, the edge cache mechanism sends the data request back toIuB DOGW (step 2230). IuB DOGW then forwards the data request toMIOP@RNC via the overlay network (step 2235). In the worst case thecontent is not cached on MIOP@RNC or MIOP@Core, MIOP@RNC routes the datarequest to via the overlay network to the MIOP@Core, which passes thedata request up the line to the internet, which delivers the requesteddata to MIOP@Core, which delivers the requested data via the overlaynetwork to MIOP@RNC (step 2240). IuPS DOGW then sends the requested datato IuB DOGW (step 2245). IuB DOGW then sends the requested data to theedge cache mechanism (step 2250). The edge cache mechanism caches therequested data (step 2255). The edge cache mechanism sends the requesteddata to IuB DOGW (step 2260). The offload data handler in IuB DOGW sendsthe requested data to NodeB (step 2265). NodeB then sends the requesteddata to UE (step 2270). At this point, method 2200 is done.

Method 2300 in FIG. 23 shows the steps performed for a cache hit in thespecific implementation in FIG. 17. The UE sends the data request toNodeB (step 2310). NodeB sends the data request to IuB DOGW (step 2320).The requested data meets the offload criteria at MIOP@NodeB (step 2330).IuB DOGW sends the data request to the edge cache mechanism (step 2340).Due to a cache hit, the edge cache mechanism sends the requested datafrom the cache to IuB DOGW (step 2350). The offload data handler in IuBDOGW sends the requested data to NodeB (step 2360). Node B then sendsthe requested data to UE (step 2370). Method 2300 shows a greatadvantage in caching data at MIOP@NodeB. With data cached at MIOP@NodeB,the data may be delivered to the user equipment without any backhaul onthe core network. The result is reduced network congestion in the corenetwork while improving quality of service to the subscriber.

The methods shown in FIGS. 18-23 provide detailed steps for the specificimplementation in FIG. 17. Other implementations may have detailed stepsthat are different than those shown in FIGS. 18-23. These are shown byway of example, and are not limiting of the disclosure and claimsherein.

The architecture of the MIOP system allows services to be layered ornested. For example, the MIOP system could determine to do breakout ofhigh-speed channels at MIOP@NodeB, and to do breakout of low-speedchannels at MIOP@RNC. In another example, MIOP@NodeB may have a cache,MIOP@RNC may also have a cache, and MIOP@Core may also have a cache. Ifthere is a cache miss at MIOP@NodeB, the cache in MIOP@RNC could bechecked, followed by checking the cache in MIOP@Core. Thus, decisionscan be dynamically made according to varying conditions of what data tocache and where.

To support the MIOP services that are possible with the mobile datanetwork 200 shown in FIG. 2, the preferred configuration of MIOP@NodeB210 is a combination of hardware and software. The preferredconfiguration of MIOP@RNC 220 is also a combination of hardware andsoftware. The preferred configuration of MIOP@Core 230 is software only,and can be run on any suitable hardware in the core network. Thepreferred configuration of MIOP@NMS 240 is software only, and can alsobe run on any suitable hardware in the core network.

In the most preferred implementation, the various functions ofMIOP@NodeB 210, MIOP@RNC 220, MIOP@Core 230, and MIOP@NMS 240 areperformed in a manner that is nearly transparent to existing equipmentin the mobile data network. Thus, the components in prior art mobiledata network 100 that are also shown in the mobile data network 200 inFIG. 2 have no knowledge of the existence of the various MIOPcomponents, with the exception of existing routers that may need to beupdated with routing entries corresponding to the MIOP components. TheMIOP services are provided by the MIOP components in a way that requiresno changes to hardware and only minor changes to software (i.e., newrouter entries) in any existing equipment in the mobile data network,thereby making the operation of the MIOP components transparent to theexisting equipment once the MIOP components are installed andconfigured. The result is a system for upgrading existing mobile datanetworks as shown in FIG. 1 in a way that does not require extensivehardware or software changes to the existing equipment. The MIOPservices herein can thus be performed without requiring significantcapital expenditures to replace or reprogram existing equipment.

Referring to FIG. 24, one suitable hardware architecture for MIOP@NodeB2410 is shown. MIOP@NodeB 2410 is one specific implementation forMIOP@NodeB 210 shown in FIGS. 2, 4 and 17. MIOP@NodeB 2410 is onesuitable example of a breakout component that may be incorporated intoan existing mobile data network. The specific architecture was developedbased on a balance between needed function and cost. The hardwarecomponents shown in FIG. 24 may be common off-the-shelf components. Theyare interconnected and programmed in a way to provide needed functionwhile keeping the cost low by using off-the-shelf components. Thehardware components shown in FIG. 24 include a system controller 2412, aservice processor 2420, a security subsystem 2430, a telco breakoutsubsystem 2450, and a fail-to-wire (FTW) module 2460. In one suitableimplementation for MIOP@NodeB 2410 shown in FIG. 24, the systemcontroller 2412 is an x86 system. The service processor 2420 is an IBMIntegrated Management Module version 2 (IMMv2). The security subsystem2430 includes an ATMEL processor and a non-volatile memory such as abattery-backed RAM for holding keys. The telco breakout system 2450performs the breakout functions for MIOP@NodeB 2410. In this specificimplementation, the x86 and IMMv2 are both on a motherboard thatincludes a Peripheral Component Interconnect Express (PCIe) slot. Ariser card plugged into the PCIe slot on the motherboard includes thesecurity subsystem 2430, along with two PCIe slots for the telcobreakout system 2450. The telco breakout system 2450 may include a telcocard and a breakout card that performs breakout as described in detailabove with respect to FIG. 17.

One suitable x86 processor that could serve as system controller 2412 isthe Intel Xeon E3-1220 processor. One suitable service processor 2420 isan IBM Renassas SH7757, but other known service processors could beused. One suitable processor for the security subsystem 2430 is an ATMELprocessor UC3L064, and one suitable non-volatile memory for the securitysubsystem 2430 is a DS3645 battery-backed RAM from Maxim. One suitableprocessor for the telco breakout subsystem 2450 is the Cavium Octeon IICN63XX.

Various functions of the MIOP@NodeB 2410 shown in FIG. 24 are dividedamongst the different components. Referring to FIG. 25, the systemcontroller 2412 implements an appliance mechanism 2510, a platformservices mechanism 2520, and an edge application serving mechanism 2530.The appliance mechanism 2510 provides an interface to MIOP@NodeB thathides the underlying hardware and software architecture by providing aninterface that allows configuring and using MIOP@NodeB without knowingthe details of the underlying hardware and software. The platformservices mechanism 2520 provides messaging support between thecomponents in MIOP@NodeB, allows managing the configuration of thehardware and software in MIOP@NodeB, and monitors the health of thecomponents in MIOP@NodeB. The edge application serving mechanism 2530allows software applications to run within MIOP@NodeB that perform oneor more mobile network services at the edge of the mobile data networkin response to broken-out data received from user equipment or sent touser equipment. In the most preferred implementation, the data brokenout and operated on by MIOP@NodeB is Internet Protocol (IP) datarequests received from the user equipment and IP data sent to the userequipment. The edge application service mechanism 2530 may serve bothapplications provided by the provider of the mobile data network, andmay also serve third party applications as well. The edge applicationserving mechanism 2530 provides a plurality of mobile network servicesto user equipment at the edge of the mobile data network in a way thatis mostly transparent to existing equipment in the mobile data network.

Referring to FIG. 26, the service processor 2420 includes a thermalmonitor/control mechanism 2610, a hardware monitor 2620, a fail-to-wirecontrol mechanism 2630, a key mechanism 2640, a system controllermonitor/reset mechanism 2650, and a display/indicator mechanism 2660.The thermal monitor/control mechanism 2610 monitors temperatures andactivates controls to address thermal conditions. For example, thethermal monitor 2610 monitors temperature within the MIOP@NodeBenclosure, and activates one or more fans within the enclosure when thetemperature exceeds some threshold. In addition, the thermalmonitor/control mechanism 2610 may also monitor temperature in thebasestation external to the MIOP@NodeB enclosure, and may controlenvironmental systems that heat and cool the basestation itself externalto the MIOP@NodeB enclosure. The hardware monitor 2620 monitors hardwarefor errors. Examples of hardware that could be monitored with hardwaremonitor 2620 include CPUs, memory, power supplies, etc. The hardwaremonitor 2620 could monitor any of the hardware within MIOP@NodeB 2410.

The fail-to-wire control mechanism 2630 is used to switch a fail-to-wireswitch to a first operational state when MIOP@NodeB is fully functionalthat causes data between the upstream computer system and the downstreamcomputer system to be processed by MIOP@NodeB 2410, and to a secondfailed state that causes data to be passed directly between the upstreamcomputer system and the downstream computer system without beingprocessed by MIOP@NodeB 2410. The key mechanism 2640 provides aninterface for accessing the security subsystem 2430. The systemcontroller monitor/reset mechanism 2650 monitors the state of the systemcontroller 2412, and resets the system controller 2412 when needed. Thedisplay/indicator mechanism 2660 activates a display and indicators onthe front panel of the MIOP@NodeB to provide a visual indication of thestatus of MIOP@NodeB.

Referring to FIG. 27, the security subsystem 2430 includes a key storage2702 that is a non-volatile storage for keys, such as a battery-backedRAM. The security subsystem 2430 further includes a key mechanism 2710and a tamper detection mechanism 2720. Key mechanism 2710 stores keys tothe non-volatile key storage 2702 and retrieves keys from thenon-volatile key storage 2702. Any suitable keys could be stored in thekey storage 2702. The security subsystem 2430 controls access to thekeys stored in key storage 2702 using key mechanism 2710. The tamperdetection mechanism 2720 detects physical tampering of MIOP@NodeB, andperforms functions to protect sensitive information within MIOP@NodeBwhen physical tampering is detected. The enclosure for MIOP@NodeBincludes tamper switches that are triggered if an unauthorized persontries to open the box. In response, the tamper detection mechanism maytake any suitable action, including actions to protect sensitiveinformation, such as not allowing MIOP@NodeB to boot the next time,erasing keys in key storage 2702, and actions to sound an alarm that thetampering has occurred.

Referring to FIG. 28, the telco breakout system 2450 includes a telcocard 2802, a breakout mechanism 2810, and an overlay network mechanism2820. Telco card 2802 is any suitable card for handling networkcommunications in the radio access network. Breakout mechanism 2810 isone specific implementation for breakout mechanism 410 shown in FIG. 4.Breakout mechanism 2810 performs the breakout functions as described indetail above. The breakout mechanism 2810 interrupts the connectionbetween the NodeB and the next upstream component in the radio accessnetwork, such as the RNC, as shown in FIG. 2. Non-broken out data fromthe upstream component is simply passed through MIOP@NodeB to the NodeB.Non-broken out data from the NodeB is simply passed through MIOP@NodeBto the upstream component. Note the path for non-broken out data is thetraditional path for data in the mobile data network before the MIOPcomponents were added. Broken-out data is intercepted by MIOP@NodeB, andmay be appropriate processed at MIOP@NodeB, or may be routed to anupstream component via a different data path, such as to MIOP@RNC viathe overlay network. The telco breakout system 2450 includes an overlaynetwork mechanism 2820 that allows MIOP@NodeB 2410 to communicate viathe overlay network. For example, MIOP@NodeB 2410 could use overlaynetwork mechanism 2820 to communicate with MIOP@RNC 220 or tocommunicate with other MIOP@NodeBs.

The edge application mechanism 2530 may provide many different mobilenetwork services. Examples of some of these services are shown in FIG.29. This specific implementation for edge application mechanism 2530includes an edge caching mechanism 2910, a push-based service mechanism2920, a third party edge application serving mechanism 2930, ananalytics mechanism 2940, a filtering mechanism 2950, arevenue-producing service mechanism 2960, and a charging mechanism 2970.The edge caching mechanism 2910 is one suitable implementation of edgecache mechanism 1730 shown in FIG. 17, and includes the functionsdescribed above with respect to FIG. 17. The push-based servicemechanism 2920 provides support for any suitable push-based service,whether currently known or developed in the future. Examples of knownpush-based services include without limitation incoming text messages,incoming e-mail, instant messaging, peer-to-peer file transfers, etc.

The third party edge application serving mechanism 2930 allows runningthird party applications that provide mobile network services at theedge of the mobile data network. The capability provided by the thirdparty edge application serving mechanism 2930 opens up new ways togenerate revenue in the mobile data network. The operator of the mobiledata network may generate revenue both from third parties that offeredge applications and from subscribers who purchase or use edgeapplications. Third party applications for user equipment has become avery profitable business. By also providing third party applicationsthat can run at the edge of the mobile data network, the experience ofthe user can be enhanced. For example, face recognition software is verycompute-intensive. If the user were to download an application to theuser equipment to perform face recognition in digital photographs, theperformance of the user equipment could suffer. Instead, the user couldsubscribe to or purchase a third party application that runs at the edgeof the mobile data network (executed by the third party edge applicationserving mechanism 2930) that performs face recognition. This would allowa subscriber to upload a photo and have the hardware resources inMIOP@NodeB perform the face recognition instead of performing the facerecognition on the user equipment. We see from this simple example it ispossible to perform a large number of different functions at the edge ofthe mobile data network that were previously performed in the userequipment or upstream in the mobile data network. By providingapplications at the edge of the mobile data network, the quality ofservice for subscribers increases.

The analytics mechanism 2940 performs analysis of broken-out data. Theresults of the analysis may be used for any suitable purpose or in anysuitable way. For example, the analytics mechanism 2940 could analyze IPtraffic on MIOP@NodeB, and use the results of the analysis to moreintelligently cache IP data by edge caching mechanism 2910. In addition,the analytics mechanism 2940 makes other revenue-producing servicespossible. For example, the analytics mechanism 2940 could track IPtraffic and provide advertisements targeted to user equipment in aparticular geographic area served by the basestation. Because data isbeing broken out at MIOP@NodeB, the analytics mechanism 2940 may performany suitable analysis on the broken out data for any suitable purpose.

The filtering mechanism 2950 allows filtering of content delivered tothe user equipment by MIOP@NodeB. For example, the filtering mechanism2950 could block access to adult websites by minors. This could be done,for example, via an application on the user equipment or via a thirdparty edge application that would inform MIOP@NodeB of accessrestrictions, which the filtering mechanism 2950 could enforce. Thefiltering mechanism 2950 could also filter data delivered to the userequipment based on preferences specified by the user. For example, ifthe subscriber is an economist and wants news feeds regarding economicissues, and does not want to read news stories relating to elections orpolitics, the subscriber could specify to exclude all stories thatinclude the word “election” or “politics” in the headline. Of course,many other types of filtering could be performed by the filteringmechanism 2950. The filtering mechanism 2950 preferably performs anysuitable data filtering function or functions, whether currently knownor developed in the future.

The revenue-producing service mechanism 2960 provides new opportunitiesfor the provider of the mobile data network to generate revenue based onthe various functions MIOP@NodeB provides. An example was given abovewhere the analytics mechanism 2940 can perform analysis of data brokenout by MIOP@NodeB, and this analysis could be provided by therevenue-producing service mechanism 2960 to interested parties for aprice, thereby providing a new way to generate revenue in the mobiledata network. Revenue-producing service mechanism 2960 broadlyencompasses any way to generate revenue in the mobile data network basedon the specific services provided by any of the MIOP components.

The charging mechanism 2970 provides a way for MIOP@NodeB to inform theupstream components in the mobile data network when the subscriberaccesses data that should incur a charge. Because data may be providedto the subscriber directly by MIOP@NodeB without that data flowingthrough the normal channels in the mobile data network, the chargingmechanism 2970 provides a way for MIOP@NodeB to charge the subscriberfor services provided by MIOP@NodeB of which the core network is notaware. The charging mechanism 2970 tracks the activity of the user thatshould incur a charge, then informs a charging application in the corenetwork that is responsible for charging the subscriber of the chargesthat should be billed.

The hardware architecture of MIOP@NodeB shown in FIGS. 24-29 allowsMIOP@NodeB to function in a way that is mostly transparent to existingequipment in the mobile data network. For example, if an IP request fromuser equipment may be satisfied from data held in a cache by edgecaching mechanism 2910, the data may be delivered directly to the userequipment by MIOP@NodeB without traversing the entire mobile datanetwork to reach the Internet to retrieve the needed data. This cangreatly improve the quality of service for subscribers by performingmany useful functions at the edge of the mobile data network. The corenetwork will have no idea that MIOP@NodeB handled the data request,which means the backhaul on the mobile data network is significantlyreduced. The MIOP components disclosed herein thus provide a way tosignificantly improve performance in a mobile data network by adding theMIOP components to an existing mobile data network without affectingmost of the functions that already existed in the mobile data network.

The mobile data network 200 disclosed herein includes MIOP componentsthat provide a variety of different services that are not possible inprior art mobile data network 100. In the most preferred implementation,the MIOP components do not affect voice traffic in the mobile datanetwork. In addition to performing optimizations that will enhanceperformance in the form of improved download speeds, lower latency foraccess, or improved quality of experience in viewing multimedia on themobile data network, the MIOP architecture also provides additionalcapabilities that may produce new revenue-generating activities for thecarrier. For example, analytics may be performed on subscriber sessionsthat allow targeting specific subscribers with additional services fromthe carrier to generate additional revenue. For example, subscriberscongregating for a live music event may be sent promotions on paid formedia related to that event. In another example, subscribers getting offa train may be sent a coupon promoting a particular shuttle company asthey walk up the platform towards the street curb. Also, premium webcontent in the form of video or other multimedia may be served fromlocal storage and the subscriber would pay for the additional contentand quality of service.

MIOP@NodeB is preferably an appliance. The difference between atraditional hardware/software solution and an appliance is the applianceinterface hides the underlying hardware and software configuration fromthe users of the appliance, whether the user is a man or a machine.Appliances for different applications are known in the art. For example,a network switch is one example of a known appliance. A network switchtypically provides a web-based interface for configuring the switch withthe appropriate configuration parameters. From the web-based interface,it is impossible to tell the internal hardware and softwareconfiguration of a network switch. The only commands available in theweb-based interface for the network switch are those commands needed toconfigure and otherwise control the function of the network switch.Other functions that might be supported in the hardware are hidden bythe appliance interface. This allows an interface that is independentfrom the hardware and software implementation within the appliance. Insimilar fashion, MIOP@NodeB is preferably an appliance with a definedinterface that makes certain functions needed to configured and operateMIOP@NodeB available while hiding the details of the underlying hardwareand software. This allows the hardware and software configuration ofMIOP@NodeB to change over time without having to change the applianceinterface. The appliance aspects of MIOP@NodeB are implemented withinthe appliance mechanism 2510 in FIG. 25.

One suitable implementation of the appliance mechanism 2510 is shown inFIG. 30. In this implementation, appliance mechanism 2510 includesmultiple appliance interfaces and multiple appliance functions. Whilemultiple appliance interfaces are shown in FIG. 30, the disclosure andclaims herein also extend to an appliance with a single interface aswell. Appliance interfaces 3010 include a command line interface (CLI)3012, a web services interface 3014, a simple network managementprotocol (SNMP) interface 3016, and a secure copy (SCP) interface 3018.The appliance functions 3020 include configuration management 3022,performance management 3024, fault/diagnostic management 3026, securitymanagement 3028, network management 3030, breakout management 3032,appliance platform management 3034, edge application management 3036,alarm management 3038, and file transfer management 3040. Additionaldetails regarding the appliance interfaces 3010 and appliance functions3020 are provided below.

The command line interface 3012 is a primary external interface to theMIOP@NodeB appliance. In the specific implementation shown in FIG. 30,the command line interface 3012 provides most of the appliance functions3020-3040, which are described in more detail below. Those commands notprovided in command line interface 3012 are provided by the SNMPinterface 3016 or the SCP interface 3018, as described in detail belowwith reference to FIG. 42.

The web services interface 3014 is another primary external interface tothe MIOP@NodeB appliance. In the specific implementation shown in FIG.30, the web services interface 3014 provides all the same functions asthe command line interface 3012.

The SNMP interface 3016 is an interface to the MIOP@NodeB appliance thatis used by an external entity such as MIOP@NMS or MIOP@RNC to receivealarms from MIOP@NodeB. For example, if a fan failed on the MIOP@NodeBappliance, a “fan failed” SNMP trap could be raised by MIOP@NodeB. Amonitor running on MIOP@NMS could catch this trap, and any suitableaction could be taken in response, including alerting a systemadministrator of the mobile data network, who could take correctiveaction, such as dispatching a repair crew to the basestation thatincludes the MIOP@NodeB appliance to repair the defective fan or replacethe MIOP@NodeB appliance. Once the repair is made, the MIOP@NMS wouldclear the SNMP trap, which would communicate to the MIOP@NodeB that therepair was made. In one specific implementation, the SNMP interfaceincludes only the functions for alarm management 3038. The SNMPinterface can also be used as a way to request and send informationbetween two network entities, such as MIOP@NodeB and MIOP@RNC, orbetween MIOP@NodeB and MIOP@NMS. However, the SCP interface 3018provides a more preferred interface for transferring data between twonetwork entities.

The SCP interface 3018 is an interface based on the Secure Shell (SSH)protocol, such as that typically used in Linux and Unix systems. SCPinterface 3018 thus provides a secure way to transfer informationbetween two network entities. The SCP interface 3018 could be used, forexample, by MIOP@NMS to transfer configuration information or softwareupdates to MIOP@NodeB. The SCP interface 3018 could likewise be used totransfer audit logs, diagnostic information, performance data, orbackups of the appliance configuration from MIOP@NodeB to MIOP@NMS.Implementing SCP is easy given the SSH already provided on MIOP@NodeBthat provides a secure shell for the command line interface 3012 to runin. In one specific implementation, the SCP interface 3018 includes onlythe functions for file transfer management 3040.

FIG. 31 shows a method 3100 for defining the appliance interfaces andfunctions for the MIOP@NodeB appliance. The appliance interfaces aredefined (step 3110). The appliance commands are defined (step 3120). Theappliance commands allowed for each appliance interface are thenspecified (step 3130). For example, the table in FIG. 42 shows for eachset of appliance functions shown in FIG. 30, which of the interfacesimplement which commands. While the table in FIG. 42 shows differentinterfaces for different commands, it is equally possible to havemultiple interfaces that implement the same command. Note the MIOP@NodeBcan include any suitable number of interfaces and any suitable number ofcommands defined on each of those interfaces.

The various appliance functions 3020 shown in FIG. 30 may be implementedusing different commands. Examples of some suitable commands are shownin FIGS. 32-41. Referring to FIG. 32, configuration management functions3022 may include breakout configuration commands 3210, edge cacheconfiguration commands 3220, platform configuration commands 3230,network configuration commands 3240, firmware/hardware configurationcommands 3250, security configuration commands 3260, and edgeapplication configuration commands 3270. The breakout configurationcommands 3210 include commands to configure the breakout mechanism inMIOP@NodeB. The edge cache configuration commands 3220 include commandsto configure caching of IP data within MIOP@NodeB. Platformconfiguration commands 3230 include commands to configure MIOP@NodeB.Network configuration commands 3240 include commands to configurenetwork connections in MIOP@NodeB. Firmware/hardware configurationcommands 3250 include commands to configure the firmware or hardwarewithin MIOP@NodeB. Security configuration commands 3260 include commandsto configure security settings in MIOP@NodeB. Edge applicationconfiguration commands 3270 allow configuring applications that run onMIOP@NodeB to provide services with respect to IP data exchanged withuser equipment. These may include native applications and third partyapplications.

Referring to FIG. 33, performance management functions 3024 may includecollect performance indicators commands 3310, counters commands 3320,and analysis commands 3330. The collect performance indicators commands3310 include commands that allow collecting key performance indicators(KPIs) from MIOP@NodeB. The counters commands 3320 include commands thatset or clear counters that measure performance in MIOP@NodeB. Theanalysis commands 3330 include commands that perform analysis ofperformance parameters within MIOP@NodeB. For example, analysis commands3330 could perform summations of key performance indicators for a giventime period.

Referring to FIG. 34, fault/diagnostic management functions 3026 mayinclude log control commands 3410, fault control commands 3420, andsystem health commands 3430. Log control commands 3410 include commandsthat collect logs, prune existing logs, purge existing logs, and setlogging parameters. Fault control commands 3420 include commands thatconfigure fault targets and view faults that have not been resolved.System health commands 3430 include commands that allowing viewingsystem health and taking actions in response to faults, such asrestarting breakout, shutdown of MIOP@NodeB, etc.

Referring to FIG. 35, security management functions 3029 include twodifferent classes of security commands, manufacturing security commands3510 and operational security commands 3520. The manufacturing securitycommands 3510 include key commands 3512, digital certificate commands3514, system state commands 3516, and hardware diagnostic commands 3518.The manufacturing security commands 3510 are used during manufacture ofMIOP@NodeB to perform security functions. The key commands 3512 includecommands to load security/encryption keys. The digital certificatecommands 3514 include commands to communicate with a trusted server tosign digital certificates. The system state commands 3516 includecommands to read and modify the state of MIOP@NodeB. System statecommands 3516 could be used, for example, to modify the state ofMIOP@NodeB from a manufacturing state to an operational state. Thehardware diagnostic commands 3518 include commands that run hardwareexercisers to verify the MIOP@NodeB is functional. The operationalsecurity commands 3520 include audit record commands 3522, which includecommands that allow reviewing and auditing records that track thesecurity functions performed by MIOP@NodeB.

Referring to FIG. 36, the network management commands 3030 includenetwork setup commands 3610, network status commands 3620, and networkdiagnostic commands 3630. Network setup commands 3610 include commandsthat setup network connections in MIOP@NodeB. Network status commands3620 include commands that allow showing network status, statistics,neighboring MIOP@NodeB systems, and current network configuration.Network diagnostic commands 3630 include commands for networkdiagnostics and tests, such as pinging an interface to see if itresponds. Note the configuration management functions 3022 shown in FIG.32 include network configuration commands, which can be used toconfigure network connections in MIOP@NodeB both during manufacturing aswell as when the MIOP@NodeB is made operational in a mobile datanetwork.

Referring to FIG. 37, the breakout management functions 3032 may includebreakout stop/start commands 3710 and breakout status commands 3720. Thebreakout stop/start commands 3710 include commands to stop and startbreakout in MIOP@NodeB. The breakout status commands 3720 includecommands to determine the state of breakout on MIOP@NodeB.

Referring to FIG. 38, the appliance platform management functions 3034may include status commands 3810, component commands 3820, healthcommands 3830, software configuration commands 3840, SNMP trap commands3840, and appliance commands 3860. The status commands 3810 may includecommands that show the health status and overload status of MIOP@NodeB.The component commands 3820 include commands that list components withinMIOP@NodeB and their versions. The health commands 3830 include commandsthat monitor the health of MIOP@NodeB, such as commands that respond tohealth and overload issues. The software configuration commands 3840include commands to upgrade or rollback software running on MIOP@NodeB.The SNMP trap commands 3850 include commands to set SNMP trapdestinations and define SNMP trap actions. The appliance commands 3860include commands to reboot MIOP@NodeB, put MIOP@NodeB to sleep for someperiod of time, and reset MIOP@NodeB to its manufacturing defaults.

Referring to FIG. 39, the edge application management functions 3036include native edge application commands 3910 and third party edgeapplication commands 3920. The native edge application commands 3910include commands to configure and manage native edge applications inMIOP@NodeB. The third party edge application commands 3920 includecommands to install, configure and manage third party applications inMIOP@NodeB.

Referring to FIG. 40, the alarm management functions 3038 include alarmconfiguration commands 4010 and alarm status commands 4020. The alarmconfiguration commands 4010 include commands to configure alarms inMIOP@NodeB. The alarm status commands 4020 include commands to determinethe status of alarms in MIOP@NodeB or to clear previously raised alarmson MIOP@NodeB. In one particular implementation, the alarm managementfunctions 3038 are available via the SNMP interface 3016. In thisconfiguration, SNMP is used by MIOP@NodeB to raise alarms that are beingmonitored. For example, if a fan failed on the MIOP@NodeB appliance, a“fan failed” SNMP trap could be raised by the MIOP@NodeB. This trapwould be caught by a monitor running on MIOP@NMS, and an alert would begiven to a system administrator monitoring the mobile data network. Thesystem administrator could then take corrective action, such asdispatching a repair crew to the basestation to repair the failed fan.Once the failure is fixed, the system administrator can clear the alarmby sending a clear SNMP trap to MIOP@NodeB.

Referring to FIG. 41, the file transfer management functions 3040include file transfer commands 4110 that allow transferring files to andfrom MIOP@NodeB. In one particular implementation, the file transfercommands 4110 are available via the SCP interface 3018. The filetransfer commands 4110 include commands in a Secure Shell (SSH), whichis a network protocol used to remote shell access to the MIOP@NodeBappliance. SSH is very commonly used for secure shell access on Linuxand Unix systems. Secure Copy (SCP) runs in SSH and allows securelycopying files between systems. The SCP interface 3018 thus provides filetransfer commands 4110 that allow transferring files to and fromMIOP@NodeB. For example, configuration files or software updates couldbe transferred to MIOP@NodeB, while audit logs, diagnostic information,performance data, and backups of the appliance configuration could betransferred from the MIOP@NodeB.

FIG. 42 shows how commands may be defined for interfaces in one specificexample. The command line interface implements all configurationmanagement commands except for file transfer commands, which areimplemented in the SCP interface. The command line interface implementsall performance management commands except for file transfer commands,which are implemented in the SCP interface. The command line interfaceimplements all fault/diagnostic management commands except for alarmtraps, which are implemented in the SNMP interface, and file transfercommands, which are implemented in the SCP interface. The command lineinterface implements all security management commands except for filetransfer commands, which are implemented in the SCP interface. Thecommand line interface implements all network management commands andall breakout management commands. The command line interface implementsall appliance platform management commands except for file transfercommands, which are implemented in the SCP interface. The command lineinterface implements all edge application management commands except forfile transfer commands, which are implemented in the SCP interface. TheSNMP interface implements all alarm management commands. The SCPinterface implements all file transfer management commands. Of course,FIG. 42 is one suitable example of specifying which appliance commandsare implemented in different interfaces. The disclosure and claimsherein expressly extend to defining any suitable number of commands onany suitable number of interfaces, including commands implemented inmultiple interfaces.

A block diagram view of the MIOP@NodeB appliance 2410 is shown in FIG.43. MIOP@NodeB appliance 2410 includes an enclosure 4310, hardware 4320and software 4330. The hardware 4320 includes network connections 4340to a downstream computer system, such as a NodeB in a basestation.Hardware 4320 also includes network connections 4350 to an upstreamcomputer system, such as an RNC. The software 4330 includes the breakoutmechanism 2810 shown in FIG. 28, and the appliance mechanism 2510 shownin FIG. 25. This simple block diagram in FIG. 43 shows the encapsulationof hardware and software within an enclosure into an appliance view,where the appliance defines one or more interfaces with commands thatare allowed to be performed on the MIOP@NodeB appliance. Creating aMIOP@NodeB appliance 2410 as shown in FIG. 43 and discussed in detailherein allows changing the implementation of hardware and softwarewithin the appliance while maintaining the consistent applianceinterface. This allows the design and functionality of the MIOP@NodeBappliance to evolve over time while maintaining the same interfaces andcommands. As a result, the MIOP@NodeB hardware and software can bechange dramatically without affecting how external components interactwith MIOP@NodeB. Of course, changes in design and improvements inperformance may give rise to new commands that could be defined in theMIOP@NodeB appliance. Note, however, that defining new commands inMIOP@NodeB would not affect the compatibility of MIOP@NodeB with othercomponents in the mobile data network that do not need the new commands.As a result, the MIOP@NodeB appliance is backwards compatible with allearlier versions of MIOP@NodeB.

FIG. 44 is a block diagram that illustrates the data paths of thefail-to-wire (FTW) module 2460. The FTW module 2460 connects a breakoutsystem (MIOP@NodeB Appliance) 2410 between a downstream computer 4410and an upstream computer 4412. In the described example, the downstreamcomputer 4410 is a NodeB 130 or Basestation 222 and the upstreamcomputer 4412 is an RNC 140 in a mobile data network as described above(See FIG. 1). The primary data path 4414 of the system is a network datacommunication signal between the upstream computer 4412 and thedownstream computer 4410. In the specific example described herein, theprimary data path 4414 is a voice/data stream connection from thebasestation 130 to the backend of a mobile data network. The FTW module2460 acts to preserve the primary data path if there is a failure in thebreakout system 2410. The FTW module 2460 provides a breakout data path4416 that routes data normally on the primary data path 4414 through thebreakout system 2410. When there is some kind of failure or problem inthe breakout system 2410 the FTW module 2460 connects the downstreamcomputer 4410 with the upstream computer 4412 through the fail-to-wiredata path 4418 on the FTW module 2460 that preserves the primary datapath. The FTW module 2460 is preferably a removable module with aconnector 4422 that connects to an edge card connector 4424 at a moduleport 4420 in the breakout system 2410.

Again referring to FIG. 44, the FTW module allows the breakout system2410 to move between the primary data path 4414 and the breakout datapath 4416. Moving between these two paths requires a temporaryinterruption of data traffic on the primary data path. This temporaryinterruption of the data traffic will be simply a small glitch that willnormally be compensated for by retransmitting of missed data packets andother failure mechanisms in the mobile data network such that thetemporary interruption will not be observable to the human user on theuser equipment.

FIG. 45 is a block diagram that illustrates the basic operation of thefail-to-wire module 2460. The FTW module 2460 operates to connect thebreakout system (MIOP@NodeB) 2410 between a downstream computer 4410 andan upstream computer 4412. These connections are made with switches4510. The switches when activated break out the primary data path toroute network signals between the downstream computer 4410 and theupstream computer 4414 through the breakout system 2410 as describedherein. The switches 4510 in this example are double poll double throwelectrically actuated switches such as a relay, electrical solenoid or areed switch. Alternatively, the switches could also be optical switchesfor optical network signals. The switches 4510 are connected such thatin the non-energized state the upstream and downstream computers areconnected through the fail-to-wire path 4418 as shown. This connectioninsures that if power is lost from the FTW module 2460 then the modulewill preserve the primary data path 4414 shown in FIG. 44. The networkdata signals 4512 of the upstream computer 4410 and the downstreamcomputer 4412 are connected to the switches 4510 of the FTW module 2460through connectors 4514. The other output of each of the switches 4510is connected to breakout system server ports 4514 of the breakout system(MIOP@NodeB) 2410. In the illustrated example only a single set ofswitches is shown that operation to switch a single network data signalpair (transmit and receive) from the upstream computer to the downstreamcomputer, however, multiple sets of switches could be configured in asingle FTW module to switch multiple network data signal pairs.

Again referring to FIG. 45, the activation of the switches 4510 isthrough a system health signal 4516 connected to a control input of eachswitch 4510. With the switches connected as described and shown in FIG.45, the FTW module 2460 provides the network connections 4512 of thedownstream computer and the upstream computer to the breakout system(MIOP@NodeB) 2410 when the switches are activated. When the systemhealth signal 4516 is not active the switch contacts are as shown inFIG. 45, and the switches route the network connections 4512 through theFTW data path 4418. The system health signal 4516 is controlled by thehealth monitor 3440 in the breakout system (MIOP@NodeB) 2410. In thespecific example described herein the health monitor is a softwaremechanism that is part of the platform services mechanism 2520introduced with reference to FIG. 25.

FIG. 46 is a block diagram that illustrates a more detailed example ofthe FTW module 2460 connected into a breakout system, in this casespecifically the MIOP@NodeB Appliance 2410. The switches 4510 of the FTWmodule 2460 are connected to the NodeB or basestation (downstreamcomputer) 130 and the RNC (upstream computer) 140 as described above.The FTW module 2460 is connected to the MIOP@NodeB Appliance 2410through an I/O adapter 4610 connected to a backplane 4612. Components ofthe MIOP@NodeB 2410 such as the system controller 2412 are alsoconnected to this same backplane 4612 so they can share data and controlsignals 4614 on the backplane 4612. In the illustrated example describedherein the system controller is an X86 processor card as describedabove. Network signals 4514 from the FTW module 2460 are connected totransceivers 4616 in the I/O adapter 4610. Outputs from the transceivers4616 are applied to a serializer-deserializer (SERDES) 4618 that is partof an adapter controller 4620 on the I/O adapter 4610. The adaptercontroller 4620 receives control input 4622 from the MIOP@NodeB 2410through an I/O controller 4624 in the adapter controller 4620. The I/Ocontroller 4624 in the adapter controller 4620 is connected to anotherI/O controller 4626 on the system controller 2412.

Again referring to FIG. 46, as described above the switches 4510 arecontrolled by a system health signal 4516 from a health monitor 3440. Inthis detailed example, the health monitor 3440 generates a controlsignal 4628 to the FTW module control 4640 which then generates thesystem health signal 4516 to the switches 4510. The FTW module control3340 may contain various electronic circuits to control the FTW switches4510. In this example, the FTW module control 4640 is controlled by thehealth monitor 3340. Further, in this example the health monitor 3440 ispart of the platform services 2520 which is a software entity primarilyexecuting on the system controller 2412. The health monitor 3440 hasinputs 4630 that originate in various systems, both software andhardware to report the health of a subsystem.

As described above, the FTW module is preferably a removable module thatconnects into the breakout system. Since the de-activated switches placethe FTW module in the fail-to-wire or bypass mode, all network dataincluding voice and data streams between the downstream computer and theupstream computer are able to remain active on the FTW module when thereis no power to the FTW module from the breakout server. This allows theFTW module to be removed from the failed breakout system or failedserver without interrupting the network data connections, which alsoallows the breakout system to serviced or replaced. When it has beendetermined that the breakout system has failed and the FTW module is inthe fail to wire mode, the FTW module can be removed from the breakoutsystem. In the basestation of a typical mobile data network the FTWmodule would be plugged into a breakout system or MIOP@NodeB housed in arack of computer equipment. The FTW module can be unplugged and thensimply hung or secured on the rack holding the breakout system while thebreakout out system is replaced with a new breakout system. The FTWmodule can then be hot plugged into the new breakout system. This meansthat the FTW module is plugged in while the network data connection onthe FTW module is still active even though the FTW module is not poweredup. The new breakout system can then be powered up and when it becomesoperational the health monitor would activate the health signal to placethe FTW module in the system network communication or breakout mode thatuses the breakout data path to route signals to the breakout system.

FIG. 47 illustrates a block diagram of an exemplary control architecture4700 for the FTW module 2460 in a breakout system 2410 as describedabove. In this example, the health monitor 3440 in the platform services2520 receives health monitor inputs 3330A-C from multiple subsystemswhich allows the health monitor to consider the complete health of thebreakout system 2410 in determining whether to enable or disable the FTWmodule 2460. In the illustrated example, the health monitor 3440receives health monitor input 4630A from subsystem A 4710, healthmonitor input 46330B from subsystem B 4712 and health monitor input4630C from subsystem C 4714. Each of the subsystems may receive inputfrom one or more control points as described below. As the breakoutsystem 2410 boots, the FTW control mechanism 2630 (introduced withreference to FIG. 26) in conjunction with the health monitor 3440 ensurethe FTW module 2460 is in the by-pass or FTW state until all the controlpoints have had their status verified. As each subsystem in the breakoutsystem initializes, the health monitor 3440 will monitor whether allrequired control points are accountable. During the initialization, somecontrol points communicate their status to the health monitor via asoftware service. In other cases, the health monitor must request thestatus of certain control points to ensure of their health and level ofinitialization. When the health monitor and the FTW control mechanism2630 have determined that all required control points are initialized orotherwise indicate a ready state, the breakout system 2410 is then readyto step into the telecommunications traffic flow. At that point, the FTWcontrol mechanism 2630 will place the FTW module 2460 in the systemnetwork communication state, thereby putting the breakout system 2410 inthe path of the telecommunication traffic flow as described above.

Again referring to FIG. 47, the health monitor 3440 gathers healthmonitor inputs 4630A-C from multiple intelligent subsystems. In theillustrated example, subsystem A 4710 is the system controller 2412described above. Similarly, subsystem B 4712 is the service processor2420 and subsystem C 4714 is the telco breakout subsystem 2450. Thehealth monitor input 4630A includes health data from various controlpoints gathered by the system controller. In this example, the controlpoints include processes 4720 executing on the central processing unit(CPU) 4722 of the system controller 2412, and status inputs from the CPU4722 and memory 4724. Processes 4720 could include such things asmonitoring data queues to insure they are draining or being processedwithin specified limits. In subsystem B 3712, the service processor 2420collects input from control points such as an operations panel 4726,tamper switches 4728, thermal indicators 4730 and fans 4732. Othercontrol points include performance metrics of the various systems. Theservice processor communicates the health monitor input 4630B to thehealth monitor 3440 over a universal serial bus (USB). In theillustrated breakout system, the telco breakout subsystem 2450 collectsinputs from control points such as the breakout process 4734 and thetelco communication process 4736. The breakout process 4734 and thetelco communication process 4736 are critical processes of the breakoutsystem 2410. The breakout process 4734 manages the breakout of datastreams of IP traffic from the voice traffic passed through the breakoutsystem. The telco communication process 4736 handles all data flowingthrough the breakout system to the upstream and downstream mobile datanetwork entities to place the breakout system as an active device in themobile data network but appear as a passive device between the RNC 140and the NodeB 130 (FIG. 1 a) as show and described above. If either ofthese critical processes detect unrecoverable failures, the healthmonitor 3440 is alerted by the telco subsystem 2450. In the illustratedexample herein, the telco breakout subsystem 2450 communicates thehealth monitor input 4630C to the health monitor over a PCIe bus on thebackplane 4612 in FIG. 46.

Again referring to FIG. 47 the health monitor 3440 communicates on thecontrol signal 4716 to the FTW control mechanism 2630 when the FTWmodule 2460 needs to change state to the FTW state. The FTW controlmechanism 2630 communicates on the control signal 4628 to the FTW modulecontrol 4640 as described above. The FTW module control 4640 generatesthe health signal 4516 that activates switches in the FTW module 2460 toput the breakout system in line with the upstream and downstreamcomputer systems as described above.

The fail to wire control system 4700 preferably includes a heartbeatmechanism 4718 that requires a periodic signal or pulse on signal 4716from the health monitor to indicate the system is operating properly. Ifthe periodic pulse from the health monitor fails a timing criteria,which indicates that the health monitor process is no longer running,then the FTW control mechanism 2630 will cause the system to enter thefail to wire state by in-activating the switches in the FTW module 2460as described above. In the illustrate example, the heartbeat mechanism4718 is a software entity in the FTW control mechanism 2630.Alternatively the heartbeat mechanism could be hardware connected to theFTW control mechanism and physically located on the FTW module control4640 or on the service processor 2420. Timing criteria that signifies afailure could include an absence of any pulse, the time between pulsesoutside a given threshold, or any other defined interruption.

During breakout system operation, the health monitor 3440 and the FTWcontrol mechanism 2630 periodically monitor the control points to ensurebreakout system optimization can continue. If one of the control pointsis unresponsive or reports an error condition or non-operational status,the autonomic recovery mechanism 3450 in the health monitor 3440 willdetermine the severity and attempt to recover from the error asdescribed further below. If the error is critical and not recoverable,the autonomic recovery mechanism will disable the FTW module as neededto remove the breakout system from the telecommunication flow tomaintain the integrity of the mobile data network. As used herein, anerror is critical if non-recovery from the error will result inadversely affecting the communication between the RNC and the NodeBbasestation as described with reference to FIG. 44. During breakoutsystem operation, some control points will communicate their status viaa software service. If recovery actions are required, for examplerestarting and reinitializing an intelligent subsystem or allsubsystems, the autonomic recovery mechanism 3450 may manageinactivating the FTW module and then again activating it whenre-initialization is complete.

FIG. 48 illustrates a high level view of the MIOP hierarchy ofcomponents. The MIOP components illustrated here are the same as thoseshown in FIG. 2. The MIOP@NMS 240 communicates with the MIOP@Core 230,one or more MIOP@RNCs 220 and a number of MIOP@NodeBs 2410. FIG. 48provides a hierarchal view of these components to illustrate that theMIOP@NMS 240 can manage a large number of MIOP@NodeB appliances 2410where the MIOP@NodeB appliances have their own autonomic error recoveryas described herein. The autonomic error recovery function of theautonomic recovery mechanism 3450 allows the MIOP@NodeB networkappliance to hide the error recovery complexities from the networkmanagement system 240 upstream in the mobile data network.

FIG. 49 is a block diagram that illustrates how the autonomic recoverymechanism 3450 deals with the different types of errors. The autonomicrecovery mechanism 3450 is part of the health monitor 3440 in theplatform services 3520 as described above with reference to FIG. 47. Theerrors received by the platform services range in severity 4910 fromlowest to highest as shown in FIG. 49. The types of errors starting withthe most severe include critical, non-recoverable 4912; critical,recoverable 4914; non-critical, non-recoverable 4916; and non-critical,recoverable 4918. When a non-critical, recoverable error 4918 occurs theautonomic recovery mechanism will attempt recovery actions 4920 asdescribed below. If the recovery actions fail 4922 the autonomicrecovery mechanism will notify the core network of the error 4924. Ifthere is a non-critical, non-recoverable error 4916, then the autonomicrecovery mechanism will notify the core network of the error 4924. Whena critical, recoverable 4914 error occurs the autonomic recoverymechanism will attempt recovery actions 4926 as described below. If therecovery actions fail 4928, the autonomic recovery mechanism will engagefail-to-wire 4930 as described above. If there is a critical,non-recoverable error 4912, then the autonomic recovery mechanism willengage fail-to-wire 4930.

The health monitor 3440 collects errors from the various subsystems asdescribed above. The autonomic recovery mechanism 3450 determines how torespond to these errors and whether to engage the FTW module 2460 asdescribed above (FIG. 47). Note that upon successful error recovery, thetypical action is to report the error, and its successful recovery, tothe MIOP@NMS system. However, depending on the severity of the error, anacceptable alternative would be to simply log the recovery and letMIOP@NMS become aware of it as part of its normal log collection andanalysis process.

Examples of critical, non-recoverable errors may include power failureof one or more systems, failure of all the fans, failure of the telcobreakout subsystem 2430, failure of the system controller 2412, failureof the service processor 2420, or activation of the tamper switches.Critical, recoverable errors 4914 may include a failure of the heartbeatmechanism 3718, a management task failure on the service processor 2420,a MIOP@NodeB cache corruption, loss of connectivity to RNC 140 or OSN170 network, loss of all virtual Ethernet devices, a thermal event,software/firmware upgrade failure, or if network admission is denied.

Examples of non-critical, non-recoverable errors could include a singlepower supply failure, a single hard drive failure, a single fan failure,a single memory DIMM failure, etc. Examples of non-critical, recoverableerrors could include third party application process failures, loss of asingle virtual Ethernet device, an application process consuming toomany resources (CPU, memory), key processes not making sufficientprogress (i.e. process appears hung), etc.

Critical, Non-Recoverable Errors. Critical, non-recoverable errors willresult in a fail to wire to ensure integrity of the mobile data network.Some, like tamper detection, will result in keys being wiped from thesystem before complete shutdown. If possible, a notification describingthe critical failure will be sent to the MIOP@NMS system so that anoperator can be made immediately aware of the FTW. Critical,non-recoverable errors typically will require human intervention forrecovery.

Critical, Recoverable Error (Example 1). The Telco breakout subsystem2450 (Cavium card) fails to respond to a watchdog timer.

Recovery Actions:

-   -   1) Restart the breakout card. If this succeeds, exit recovery.        If this fails, proceed to step 2.    -   2) Restart system controller. If this restores communication        with the breakout card, exit recovery. If this fails, proceed to        step 3.    -   3) Notify MIOP@NMS of non-recoverable failure (if possible).

Critical, Recoverable Error (Example 2). Software/firmware upgradefailure

Recovery actions:

-   -   1) Retry upgrade. If this succeeds, exit recovery. If this        fails, proceed to step 2.    -   2) Restart component (edge application, security, messaging,        etc) and retry upgrade. If this succeeds, exit recovery. If this        fails, proceed to step 3.    -   3) Restart subsystem (Cavium card, telco card, x86, etc.) and        retry upgrade. If this succeeds, exit recovery. If this fails,        proceed to step 4.    -   4) Roll back to previous software/firmware level. If this        succeeds, notify MIOP@NMS of the failed upgrade but continue to        operate and exit recovery. If this fails, proceed to step 5.    -   5) Notify MIOP@NMS of non-recoverable failure (if possible).

Critical, Recoverable Error (Example 3): Thermal event. This exampleuses a combination of hardware, software, and the larger telco networkenvironment to attempt recovery.

Recovery actions:

-   -   1) Turn up fan speed. If this succeeds in alleviating the        thermal condition, exit recovery. If this fails, proceed to step        2.    -   2) Reduce number of users that have broken out traffic, thus        reducing overall CPU and memory workload. If this succeeds in        alleviating the thermal condition, exit recovery. If this fails,        proceed to step 3.    -   3) Reduce/degrade/shut down 3rd party applications (in priority        order) to reduce workload. If this succeeds in alleviating the        thermal condition, exit recovery. If this fails, proceed to step        4.    -   4) Contact other MIOP@NodeBs to see if offloading some of the        workload is possible. If this succeeds in alleviating the        thermal condition, exit recovery. If this fails, proceed to step        5.    -   5) Contact the MIOP@RNC to reduce traffic to this MIOP@NodeB. If        this succeeds in alleviating the thermal condition, exit        recovery. If this fails, proceed to step 6.    -   6) Perform channel stitching to move any broken out PDP contexts        to MIOP@RNC or MIOP@Core. Once this is done, to the extent        possible, quickly but as gracefully as possible shut down prior        to fail-to-wire.    -   7) Notify MIOP@NMS of non-recoverable failure (if possible).

Non-critical, non-recoverable errors.

Non-critical, non-recoverable errors will result in a notification tothe MIOP@NMS. Most non-critical, non-recoverable errors are hardwarefailures that result in a loss of redundancy or force the MIOP@NodeB torun at a reduced capacity (such as in the case of a loss of one diskdrive or one memory DIMM). Typically these errors will require humanintervention to fully recover from, but they are not an immediate issuesince the MIOP@NodeB is able to partially recover and still operate at areduced capacity.

Non-critical, non-recoverable error (Example 4). Loss of a memory DIMM(but some DIMMs still active)

Recovery actions:

-   -   1) If the loss of the DIMM causes the MIOP@NodeB software to        become unstable (this is likely) then reboot the MIOP@NodeB        system and proceed to step 2. If the software is able to        continue operating through the loss of the DIMM, then proceed        directly to step 2 with no reboot.    -   2) Evaluate how much memory capacity has been lost and adjust        workload accordingly.        -   A. If half the memory is lost, then reduce the max number of            contexts that can be broken out accordingly.        -   B. Selectively disable/end edge applications (in priority            order) based on available memory.        -   C. Reduce the frequency of certainly automated tasks, such            as performance data collection so that there are fewer time            windows where system tasks are consuming precious memory            resources.    -   3) Notify MIOP@NMS of the non-critical, non-recoverable failure,        but continue to operate.

Non-critical, recoverable errors (Example 5). Third party applicationfails

Recovery actions:

-   -   1) Restart 3rd party application. If this succeeds, exit        recovery. If this fails and this is a high priority application        (e.g., cache), proceed to step 2. Otherwise, proceed to step 3.    -   2) Restart application in a fresh guest container such as a        kernel-based virtual machine (KVM). If application successfully        restarts, exit recovery. If this fails, proceed to step 3.    -   3) Disable application and reclaim resources (CPU, memory,        etc.).    -   4) Notify MIOP@NMS of non-critical, recoverable failure.

Non-critical, recoverable errors (Example 6). Loss of connectivity of asingle virtual Ethernet adapter.

Recovery actions:

-   -   1) Reset virtual Ethernet adapter. If this succeeds, exit        recovery. If this fails, proceed to step 2.    -   2) Destroy and recreate virtual Ethernet adapter. If this        succeeds, exit recovery. If this fails, proceed to step 3.    -   3) Restart the x86. If the virtual Ethernet adapter can        successfully be created, exit recovery. If this fails, proceed        to step 4.    -   4) Notify MIOP@NMS of non-critical, recoverable failure.

Non-critical, recoverable errors (Example 7). Application using too manyresources (memory, CPU, etc).

Recovery actions:

-   -   1) Restart the application. If application uses only the        expected resources, exit recovery. If this fails, proceed to        step 2.    -   2) Query the MIOP@NMS system to see if there is by chance a new        fix available for the application. If there is, download and        install the new application. If the new application installs and        starts correctly, exit recovery. If this fails or no update is        available, proceed to step 3.    -   3) Disable the application.    -   4) Notify MIOP@NMS of non-critical, recoverable failure.

FIG. 50 is a flow diagram of a method 5000 for the autonomic recoverymechanism to provide autonomic recovery from a variety of errors in abreakout appliance at the edge of a mobile data network. The autonomicrecovery mechanism processes errors received by the breakout system. Thesteps of the method 5000 are preferably performed by the autonomicrecovery mechanism but may also be performed by other parts of thebreakout system such as the platform services. If the error is anon-critical, recoverable error (step 5010=yes) then attempt recoveryactions 5012. If the recovery action is not a failure (recoverysuccessful) (step 5014=no) then the method is done. If the recoveryaction is a failure (not successful) (step 5014=yes) then notify thecore network of the error (step 5016) and the method is done. If theerror is not a non-critical, recoverable error (step 5010=no) then go tostep 5018. If the error is a non-critical, non-recoverable error (step5018=yes) then notify the core network of the error (step 5016) and themethod is done. If the error is not a non-critical, non-recoverableerror (step 5018=no) then go to step 5020. If the error is a critical,recoverable error (step 5020=yes) then attempt recovery actions 5022. Ifthe recovery action is not a failure (recovery successful) (step5024=no) then the method is done. If the recovery action is a failure(not successful) (step 5024=yes) then fail-to-wire (step 5026) and themethod is done. If the error is not a critical, recoverable error (step5020=no) then go to step 5028. If the error is a critical,non-recoverable error (step 5028=yes) then fail-to-wire (step 5026) andthe method is done. If the error is not a critical, non-recoverableerror (step 5028=no) then go to a null or undefined state error routine(step 5030) and the method is done.

FIG. 51 is a flow diagram of a method 5100 for the autonomic recoverymechanism to attempt recovery actions. Method 5100 is one possibleimplementation of steps 5012 and 5022 in method 5000 to attempt recoveryfrom an error. The steps of the method 5000 are preferably performed bythe autonomic recovery mechanism but may also be performed by otherparts of the breakout system such as the platform services. First,perform an appropriate hardware recovery action to overcome the error(step 5110). Determine if the recovery is successful (step 5120). If therecovery action is successful (step 5120=yes) then the method is done.If the recovery action is not successful (step 5120=no) then attempt anappropriate software recovery action (step 5130). Determine if therecovery is successful (step 5140). If the recovery action is successful(step 5140=yes) then the method is done. If the recovery action is notsuccessful (step 5140=no) then attempt an appropriate network recoveryaction (step 5150). The method is then done.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language, StreamsProcessing language, or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The methods disclosed herein may be performed as part of providing aweb-based service. Such a service could include, for example, offeringthe method to online users in exchange for payment.

The disclosure and claims are directed to a mobile data network thatincludes an appliance that performs one or more mobile data services inthe mobile data network. The appliance includes a mechanism whichprovides autonomic recovery for a breakout appliance at the edge of amobile data network from a variety of errors using a combination ofhardware, software and network recovery actions. The error recoveryfunctions are within a network appliance to hide the error recoverycomplexities from the management system upstream in the mobile datanetwork.

One skilled in the art will appreciate that many variations are possiblewithin the scope of the claims. Thus, while the disclosure isparticularly shown and described above, it will be understood by thoseskilled in the art that these and other changes in form and details maybe made therein without departing from the spirit and scope of theclaims. For example, while the mobile data network in FIG. 2 anddiscussed herein is in the context of a 3G mobile data network, thedisclosure and claims herein expressly extend to other networks as well,including Long Term Evolution (LTE) networks, flat RAN networks, andcode division multiple access (CDMA) networks.

1. A method for operating a breakout system for communicating with abasestation that transmits and receives radio signals to and from userequipment, wherein the basestation is part of a radio access networkthat communicates with a core network in the mobile data network, thebreakout system having an enclosure, the method comprising the steps of:providing a fail-to-wire (FTW) module with a primary network data paththat connects an upstream computer to a downstream computer and abreakout network data path that connects the upstream computer systemand the downstream computer system to the breakout system; providing aplurality of switches that switch between the primary network data pathand the breakout network data path, wherein the switches in theinactivated state preserve the primary data path and in the activatedstate route input connections from the upstream computer and thedownstream computer on the breakout data path to the breakout system; acontrol input to the switches driven by a system health signal thatactivates the plurality of switches to connect the breakout data pathwhen the breakout system is operational; receiving a health inputs froma plurality of intelligent subsystems of the breakout system todetermine errors in the breakout system; and recovering from an error inthe breakout system using a combination of hardware recovery actions,software recovery actions and network recovery actions, and where theerror is critical and non-recoverable the autonomic recovery mechanismdrives the health signal to activate the plurality of switches andremove the breakout system from the mobile data network.
 2. The methodof claim 1 wherein the software recovery actions include at least one ofthe following: restarting an application, querying a network managementsystem to determine if there is an update available for a failingapplication and re-installing the application, and disabling theapplication.
 3. The method of claim 1 wherein the hardware recoveryactions include at least one of the following: restarting a subsystem,controlling environmental systems, reducing users on broken out trafficon the breakout system, adjusting workload of the breakout system. 4.The method of claim 1 wherein the network recovery actions include atleast one of the following: requesting the network management system toreduce the traffic to this breakout system, and offloading work to aneighboring breakout system.
 5. The method of claim 1 wherein thesubsystems of the breakout system receive input to determine the errorsfrom control points chosen from the following: processes executing onthe system processor, CPU status, memory status, operations panelstatus, tamper status, thermal status, fans status, performance metrics,breakout system status and telco network status.
 6. The method of claim1 wherein the step of recovery from an error using the hardware recoveryactions, the software recovery actions and the network recovery actionsis done on a sliding scale depending on the severity of the problem tominimize disruption to traffic flowing through the breakout system. 7.The method of claim 1 wherein the breakout system is an appliance. 8.The method of claim 1 wherein the error is critical and recoverable, butrecovery from the error failed.